DOCUMENT RESUME 



ED 236 159 

AUTHOR 
TITLE- 

INSTITUTION 

SPONS AGENCY 
PUB DATE 
CONTRACT 
NOTE' 

PUB TYPE 

EDRS. PRICE 
DESCRIPTORS 



IDENTIFIERS 



TM 830 433 

Estes, Gary D. ; * Demaline , Randy 
Phase II: Final Evaluation Report of Minimum 
Competency Testing Clarification Process. 
Northwest Regional Educational Lab, , Portland, 
Oreg. 

National Inst, of Education (ED) , Washington, DC, 
Jan 82 
400-80-0105 

199p.; Appendices A-D contain small print. For 
related documents, see ED 214 999-215 002. 
Reports - Evaluative/Feasibility (142) 

\ i 

MF01/PC08 Plus Postage. 

^Audience Analysis; Elementary Secondar^xJSducation; 
^Evaluation Methods; Hearings; Informatit 
Dissemination; *Minimum Competency Testing; "Policy 
Formation; *Program Evaluation; Quest ionnaires f\ 
Testing Problems; Videotape Recordings 
♦Adversary Evaluation Model; *Judicial Evaluation 
Approach / 



ABSTRACT 



r " On July 8 f 9 and 10, 1981 the National" Institute of 

Education (NIE) sponsored a "hearing'! on minimum competency testing 
(MCT). Thehearing, which was to clarify issues related to MCT, 
mddeled after adversarial and judicial evaluation approaches. 



Northwest Regional Educational Laboratory was responsible 
overall evaluation which was divided into two phases: a p 



was 
The J 
for the 
process 

evaluation and an outcome evaluation of \ the Clarification Process. 
This report incorporates findings from Phases I and II., It appears 
thai: the Clarification Process is a/ viable approach for studying 
major education or policy issues. Tha information produced from this 
application appears to be most useful, for] summative or general * 
information purposes. It is recommended tjiat future studies or 
efforts attempt to begin addressing the extent to which the 
adversarial nature of the Clar if ical^ion Process, rathear than 
individuals' testimony and videotapp presentations , contributed to 
the positive evaluations of the Clarification Process. (BW) 
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' EXECUTIVE SUMMARY 

On July 8, 9 and 10 , 1981 the National Institute of Education (NIE) 
sponsored a "hearing" pn minimum competency testing (MCT). This hearing 
represented 10 months of study and interviewing by two teamsr-one - 
assigned to argue for MCT and one to argue against MCT. The hearing 
which was to clarify issues related to MCT was labeled the MCT 
Clarification Process and was modeled after adversarial and judicial 
evaluation approaches. The NIE was interested in the approach as a 
viable alternative to mere traditional studies largely as a result of the 
potential to involve constituencies and to provide infontiation to state 
or local agencies without a federal agency representing a particular 

.... 1 0 

position oh MCT. The NIE was also interested in evaluating the 
effectiveness of this approach and commissioned this evaluation. 

The Northwest Regional Educational Laboratory was responsible f y or the 
overall evaluation which was divided into two phases. Phase I watf-a 
process evaluation and Phase II was an outcome and overall evaluation of 
the Clarification Process. The Phase II; Fin^l Ev aluation Report 

"Mi- 

incorporates findings from Phases I and II. It is an intent that these 
findings will be helpful to others undertaking similar efforts. 

The primary audience for this report is the NIE since they were the 
sponsor and called for the evaluation. We see them as interested in an 
external view of how the process proceeded and how the products were 

received. A primary use for this evaluation is an aid to deciding 

i , / 

whether future activities like this will be beneficial in other areas; / 
S " • . - ■ J * /' •' 

It will also point out aspects which might be useful in other 

: . - ' . / \ 

applications and weaknesses that can be anticipated. / 



Secondary audiences include those agencies, public and private, who 
might be considering alternative evaluation tools. These include states 
and local school districts and professional evaluatprs interested in 
alternative techniques which might mee£ evaluation needs better thai 
other more frequently used approaches. 

This ^valuation includes the process cif developing pro and con cases 
and the outcomes which consist of the hearing, videotapes and written 
materials. Observations at case development meetings, reviews of case 
development documentation and interviews with Clarification Process 

participants were data sources for the process eValuation. Hearing data 

t 

sources were a questionnaire administered at the end of the hearing, 
observations oi the hearing and interviews with individuals attending the 
hearing. The videotapes and written materials were evaluated through 
questionnaires and open-e?nded discussions in state meetings with state , 
level and local district staff. The states were California, Florida, 
Illinois, Montana, Texas and Wisconsin. 

The remainder of this summary highlights findings from the evaluation. 
Process Findings 

*> A project such as the Clarification Process should involve an 
- advisory or constituency group injdesigning the study, ^ 
recommending participants and suggesting issues as was done in 

\ ■ ■ i 

\ this case. 

\ • Team compositions affect the roles team members play, ease of 
communication and resources needed. 
• It is likely that slightly different cases will be developed for 
pro and con sides of issues. 



VI 



Framing and fleliftcting of issues need to bo mora efficient and ( 
less tiDQ consuming with increased time given to the case 
development. The work done in the Clarification Process and 
papers provided by the team leaders will facilitate issue 
selection in future applications. 

Tension in the Clarification Process case development increased 
when the hearing neared. Teams 1 , position statements, plans for 
direct examination and cross-examination should be delivered on 
time to reduce tension and improve responsiveness of cases to 
each other. It is also recommended ttiat tension be anticipated 
to minimize negative effects on the participants or process. 
It is recommended thit definitions and stipulated agreements be 
reviewed in greater details. If it is not possible to' introduce^ 
stipulated agreements through direct or cross-examination, it is 
important to review ,the reasons the teams agreed to the stipulated 
agreements. . 

A shortened hearing of one or two days with early invitations to 
attend the hearing will facilitate greater interest and 
participation* Additionally, regional hearings could be 
sponsored if audience participation is important. 
In future applications, it would be desirable to have a training 
pack&ge consisting of a one- or V two-hour edited videotape, 
written materials and trainer's guide, and .more resources for a 
dissemination plan which included providing' technical 
assistance-type sessions fog intended audiences. 



\ 
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Outcome Findings 

It appoars the Clarification Process is best described as having 
synthesized information which ifl helpful in understanding and 
discussing MCT. 

• Information from the Clarification Process was rated more useful 
for general information or discussion than for program 
implementation or revision decisions. The User's Guide was more 
useful than the videotapes for the latter. General interest in 

> using the information was expressed. 

• Audiences felt that a variety of viewpoints was presented during 
the hearings, and that a fair debate of the issues was 
provided. The prb team's case was rated slightly stronger than 
the con team's in dlarity of presentation. Audiences also felt 
that comprehensive cases were provided by the pro and con teams, 



with the pro team's case viewed as more comprehensive. 



The use of individual^ testimony in presenting MCT issues was 
rated a strength of the Clarification Process. It might be 

1 • ~ * in otner st 



possible to incorporate these features 
evaluations. 



studies of 



• Individuals indicated that bilingual education, school finance, 

v • . \ ' | ' 

tax tuition credits, curriculum issues, special education and 

mainstr earning, and competency based education are other issues 

e. ,« 

which might be addressed by the Clarification Process. 
Conclusions \ 

It appears that the Clarification Process is a viable approach for 

\ ' 

studying major education or policy issues. The information produced, from 
this application appears to be most useful for summative or general 
information purposes. 
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It ifl recommended that future atudiee or efforte attempt to begin 
addressing tha axtant to whioh tha adv'eraarial natura of tha 
Clarification Proaeaa, rathar than individuala' taatimony and vidaotapa 
proaentationa, oontributad to tha positive evaluatione of tha 
Clarification Prooaoa. It waa not clear in fchia evaluation that tha 
poaitiva outcomes ware dapandant upon tha advaraarial <aapect of tha 
Clarification Proca8s or wara mora a function of tha vidaotapaa and 
individuala' taatimony. Studies in this area could enhance evaluation 
utilization and impact. , 

Finally, it ia possible that reduced dayo for a hearing, ahorter 
edited tapes/ fewer witneaaea presenting more in-depth testimony and more 
dijreot dissemination to a narrower audience will result in a less costly 
effort with similar or greater impact. j 
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CHAPTER I 
INTRODUCTION 



Background and Purpose for Clarification Project ~ - y: s 

^ The National Institute of Education (NIE) sponsored a project 
defeign^d to clarify issues related tQ Minimum Competency Testing (MCT). 

The MCT Clarification. Process was adapted from adversary evaluation 

* ■ • . *■ 

methodologies, in particular the judicial evaluation model. A brief 

" ■ - ■ <% ~ ■ . ' * 

background to the Clarification Process its purposes and objectives as 

outlined by NIE are given below. \ . " 

The ,NIE initiated a study in 1978 of MCT programs. The first part of 

the study^was/ to^ollect descriptive information on the MCT programs 

across the cpuxjjtry , resulting in program descriptions and typologies of 

program characteristids. The documents which were produced .provided 

descriptiveli^omation ori' factors such as whether 'tests were locally or 

state developed; whether tests were, used for grc^uation, promotion/ or 

retention r whether remedial programs were mandated; and areas in which 

■ , ■ ■■ v ' \\ ' 

tests were administered (Gorth, 1980). Although these data were valuable 

; . i ■ . ■ . ... . \ 1 ■ 

resources in characterizing MCT programs^ it was not the intention of 

• ' \ •. • ' . • \ ■ - v 

Phase I to evaluate or study major issues related t!o.MCT programs and 

their policy or pTO^ammatic implications. It was too early in the MCT 

programs to undertake\an evaluation of their effects. Phase II was 

initially intended to beNi- three-year evaluation. NIE determined that an 

evaluation of MCT as a good 6rO>ad phenomenon was not responsive to the 

current information needs or a most^a^pr^riate role for a federal 

agency*. An appropriate role was to address^th^major concerns about MCT 

programs. *** These included (a) to identify the ^aj or issues-related to MCT 



programs, and ,(b) to^rovide information which might be useful to 
individuals concerned jvit^h MCT program issues. — ' . 

The NIE (Herndon, 1980) identified three concerns which guided the 

... : \; '. . • " - j ■ 

planning of the Clarification Process. - ■ 

■ \ .'" . 

\ > ■ - 

1. In spite of the rapid and continuing growth of Minimum 

/ ■ . ■ - - • '.• ■ 

Competency Testing/ the quality of information available to 

9 ■ / I ..... 

f ' • * 

' decision makers is limited and tends to be^clouded by the_ 

; . ; '. . . \ . •" . . r 

complexity of the issues, complicated by different and con^eting 
-"interests. y . S \ V " 
' 2. ^/in order to clarify these ^issues , it is essential that the 
framework for the study recognize the social and political 

/ ". w • . * . \ ■" K ' ; . 

context of MCT and provide a structured forum for the 

, * ; ~. ■ ' ■ 

. ^presentation and examination of different perspectives on the 

issues. _ j 

3. In order for the results of the studies to be meaningful and 
valid, it is essential that botl} the decision making audience 
and other vested interest groups\have an integral role in its 
design, implementation and dissemination. 
A traditional evaluation approach was not viewed as directly 
addressing theses concerns. The Judicial Evaluation methods described by 
Wolf (197?) offered many strengths compared to the traditional evaluation 
process. Specific advantages cyted by Herndort (1980) were that the 
judicial<process : 

(a) provides a public forum ^>r\^he examination of these issues from 
different perspectives; 

(b) allows, for public participation in the process through the 
presentation of testimony;. 



(c) permits introduction of a wide range of evidence (documentary 
evidence, . human testimony , quantitative* data) , the clarification 
of which can occur immediately through cross examination, 
rebuttal testimony and the like? \ 

(d) provides a forum for including the perceptions, opinions and w , 
judgments of those affected by policy and program decisions. . 
through the use of human testimony. Often the. more "'subjective 
forms of evidence i^an help put facts into prope* perspective. 
Testimony can then be examined within the context of facts and 
situations^ ... * 

Thus, a. variation of the judicial process was adopted to examine and 
clarify the issues related to Minimum Competency Test^^^ of^t^-. 
major variations to the traditional ^udi^l^eva^uation' approach was the « 
decision not to have a jury or judge render a decision. Instead, the 
process was intended to provide information to others! who could serve 
that role in their own policy or program deliberations. 

Several products were initially intendedHojf^ull: from the 

Clarification Process (Herndon, 1980). These included (a) a videotape of 

« ' * t' • 

the entire hearing; (b) a four-hour edited videotape of the hearing; and 
(|c) a one-hour summary videotape narrated by a professional commentator. 
Additionally, written materials were intended to provide documentation 

and/overviews of information presented at the hearing. Subsequently, the 

/ 



products front, the hearing and Clarification Process in general were 

■/•■ . "■ ; ■ ■■• . . '4 ■ ■ • >'• 

revised to include: (a) -three one-hour edited videotapes, eaten -tape 
representing one day of the hearing, (b) a one-hour documentary videotape 
providing an overview of Minimum Competency Testing and the, Clarification 
Hearing, and (c) a user's guide to accompany the tapes. Additional 



materials which are available £or use from the Clarification Process are 
the coiqplete videotapes of the hearing and transcripts of the hearing. 
These materials 0 are directly available from the NIE. 

Finally, the actual hearings were viewed to be a product of the 
Clarification Process. Although the edited tapes were intended as the 
primary outcomes, it was felt that the actual hearing was a potential 
source of information for the Clarification Process audiences. 

The materials and information from the Clarification Process were 
targeted for a broad audience. Herndon (1980) indicated that the project 



was. 



to provide decision makers and other interested 

audiences., a clear understanding of the dynamics of MCT, and 

to provide a vehicle for the involvement of parents, 

teachers, students, citizen groups, administrators, school 

board members, legislators and other interested parties in 

.a process that \ help inform policy at- the state and 

local level. ' , 

• . - ■ ■ "".«'■ ■ ■' ■.■ ' ' . , 

Shoemaker (1980) more specifically identified the. major audiences to 

be (a) state and local policy makers such as school board members^— .. 

••• ■ - i 

legislators and school administrators; (b) teachers and administrators 
who must implement MCT; (c) students who must pass MCTs; (d) parents 



whose children must pass MCTs; (e) taxpayers, the general public and 
other communities who are interested in the achievement of students; 
(f) researchers who are interested in educational policy, testing issues 
and design,\ and curriculum; (g) prof essional organizations^ representing 

V ■ ■ v - ■ . m 

the interest of teachers and students. It was noted, that tras federal* 

- * " \ ". . - r • . . . / 

government wa!s not a major audience for the study. 



The above comments are intended to provide a brief background and 
overview of the Clarification process which led to a three day hearing in 
which three major issues were to be clarified: , ' 

1. Will MOT programs that use test results for gtuderft 

certification and/or classification have beneficial or harmful 



effects on students? 



.2. Will MCT programs that use 'test results for student \^ 
certification and/or; classification have beneficial or harmful 
N "\^ influences on curriculum and teaching? • 

3. Will MCT programs that use test results )for student 

certification and/or classification have positive or negative 
effects oil public perceptions of educational quality? 
Thus, the hearing and related videotapes and materials were intended 
to clarify concerns related to these issues for the audiences described 
above* 

This report is the^second phase of the MCT Clarification Process 
evaluation. This report summarizes the findings from the Phase X Process 
Evaluation and findings from the outcomes evaluation. Comments and 

• c ' 

... r> s 

recommendations for future applications are made based on findings fr,om 
the evaluations.. A more complete description and evaluation of the 
Clarification Process is contained in the Phase I report by Bourexis 
.(1981). , 

Evaluation Questions ■ " / . 

The Clarification Process represented a variation from the 
traditional approaches taken for federal j^ztiicy or evaluation studies. 
Thus, the NIE was interested in assessing the effectiveness of the 



Clarification Process. Shoemaker (1980). outlined the major "objectives 

'■■ "' ' ' ' ■ . 

for the Clarification Process: 

(a) engage those affected by MCT and tra'ae 1 who make 
decisions about MCT in a public forum to examine issues 
related to MCT; engage them in the .design and study; • ? 

provide them *n opportunity which they may not have had 
before to expresb their views/opinions and experiences; ^ ; 
(b) contribute to the information base on MCT by organizing 
information around 3-4 distinct issues (or topics) and by 
making opposing sides address' the same issues and 
/3ubissues; and (cj contribute to the information base on 
MCT by obtaining impact information not now readily 
available and by identifying both alternatives to MCT as 
well as exemplary MCT programs. . \ • 

Given those objectives for the Clarification Process, Shoemaker 

M1980) : ^ S ,Q provided an initial list of evaluation questions organized 

a^und three phases: monitoring pre-hearing activities , monitor ing 

post^hearing activities and determining the effectiveness of the project 

s for the tylE. Pre-hearing^guestions basically centered afouiid the , 

processes used in developing the cases which led to the hearing and 

subsequent materials. Pcvrt-hearing questions centered'' on assessing 

whether the information- produced from the hearing was useful, whether 

materials reached the right audiences and how the materials were being 

used. The ef f ectiveness questions related to. what improvements n^aht be 

made in the process. 

' Five questions were agreed upon by the NIE and NWREL^as the focus for 
this evaluation: 

1. How appropriate were the format and structure of the 
Clarification Process in presenting MCT issues, i.e., in what . 
ways did the Clarification Process help or hinder the presenting 
of MCT issues? * ' 

2. Did the information presented represent a fair u diversity of , 
viewpoints on each issue that was seen as clarifying and • 
illuminating? 



/3. Did the information presented add to the current understanding 



/ and knowledge of .Minimum Competency Testing? 



4. Did the audiences perceive the information to be useful in terms 
of pending policy or program decisions? 

What is the viability of this approach for other NIE efforts? 
This document reports information related to these evaluation } 
questions. The audience for the report is intended to be the NIE and 
others who might be interested in adopting or adapting this or similar 
approaches for future studies of education or policy issues. secondary 
audience is thpse interested in the substance of the MCT issues \ 
deliberated in the Clarification Process. 
Evaluation Criteria/ 



Lug 



/ 



A" discussion of the criteria os^the framework used to addresis the : 
evaluation /Questions listed above will be helpful in reading or r 
'interpreting the /results of this evaluation. Specifically, the, questions 
"above basically focus on the utility .of the Clarification Prccsss as an v 
approach for studying MCT\Lssues. ^he framework by which one judges the 

' • / ■ • . V • ■ .• : - 

utility of an evaluation or study can substantially influence the degree v 

-.. ■ ■ . • . . ■ y- '■ < - ' • ■ ■ * 

to which it is judged to be useful or not. One concept" which might be 
proposed is that there needs to be direct observable evidence : that 
information from an evaluation or study\ directly influences program or 
policy decisions. This- view might be characterized as reflecting a 
rational or systematic' approach to decision making. This approach risks 
an insensitivity to the reality of decision making in which evaluation or 
study results will be only one^6f several forms of input. Obvious other 



key factors include political concerns , logistical, constraints and 



interpersonal variables. Patton (1978) in describing the results of , 

studies assessing the utility of federal program evaluation stated: 

None of the impact described is. of the type where new 
findings from an evaluation led. direct JLy and iraaediately to 
the making of major , concrete program decisions. More ^ 
typical impact is one where that evaluation f indings 
provided additional pieces of information in a difficult 
puzzle of program action, permitting seme reduction in the 
uncertainty within which any federal decision maker 
inevitably operates. 

The orientation used in. this evaluation of the MCT Clarification Process 

is similar to that described Above by Patton (1978) and is consistent 

with the perception of AlkAn et al. (1979) , who of f er that^evaluation 

often has "incremental" rather tJSmr* w .major w influences in decisions. 

Specifically the criterion used In judging .the utility and information 

value of the Clarification Process wa6 the degree to which it appeared to 

facilitate, influence, or modify the perceptions or idea's of the intended 

users. It might be helpful to outline explicitly a criterion that was 

not applied. A restrictive criterion that was discarded in judging the 

effectiveness of the Clarification Process information was the necessity 

of evidence demonstrating that the information from the Clarificatipn 

Process was, directly applied in making decisions about MCT. If evidence 

were available that the information was useful in discussing or 

deliberating MCT issues or in making decisions, then the Clarification 

Process would be judged by us to have an impact and utility. 

The above statements are not intended to diminish "the importance for 

the Clarification Process to provide valid reliable objective data. 

These certainly were issues of concern but were not judged to be the ; 

paramount criteria for evaluating the Clarification Process. Stuff lebeam 

(1974) outlines criteria which Wolf (1979) suggested might be useful for 

evaluating adversarial approaches such as the Clarification Process. 



These are divided into four technical criteria? internal validity, 

external validity, reliability and objectivity and five utility 

n • - . 7 

v < . ■ / ■ . ■ ■ • ■ • 

criteria: relevance, importance, scope, credibility and timeliness* 
Although' these criteria were not /explicitly used to evaluate the 

Clarification Process, they were useful in formulating questions and 

\. . ' / ° • ' ■ 

identifying concerns to be addressed* 

. •• - / • ' e , ■ ' • . ■■ . '< 

A hypothetical example might assist to clarify the position used in 

■ ■ . / ' ■ / . ■ * - . 

conducting this evaluation/ It is possible that the audiences would find 

the information from the Clarification Process to be highly subjective 

■ / - " /■ 

and/not generalizable beyond the specific sites discussed in" the 
testimony or documentation. "If the issues of objectivity and validity 
were of paramount importance, then the Clarification Process would, have 
to be evaluated as/weak due to the subjectivity and 4 limited 
generalizability/ i.e., external validity. If , however, the . r . « 

Clarification Process information assisted audiences to identify MCT 
issues of concern, or to broaden their perspective through the differing 

/ • • • • - ■ ' ■ • •• 

viewponts, ,the Clarification Process might be judged to be an effective 
means for/ facilitating decision making an^ providing information. This 

positive judgment would be justifiable in our opinion, even though the 

I . "' mi ' '. . . . 

criteria of objectivity and validity as defined by Stuff lebeam (1974) 

/ ' ■ 

were not fully met. ',*.,* 

Report Organization k ' 

The methodology and approach used in addressing the evaluation 
questions are contained in Chapter II. Chapter III contains a summary of 
the findings of the Phase I Process Evaluation which was a primary 
vehicle for addressing evaluation Question i; i.e., how appropriate were 
the format and structure of the Clarification Process in presenting 



Minimum Competency Testing issues? Chapter IV provides a summary of 
reactions to the Clarification Process by audiences at the hearing and in 
six states where the videotapes and written materials were reviewed. 
Chapter V provides recommendations and conclusions based upon findings 
from the process and outcomes evaluation. ^ 



CHAPTER II. 
METHODOLOGY 

Process Evaluation of Case Development 

The primary approach used in monitoring the case development for the 
Clarification Process was attendance at case development meetings. Pat 
Bourexis and/or Gary Estes of the evaluation team attended each o£. seven 
meetings, beginning with a training session in Washington, D.C. in 
October, 1980 and concluding with the pre-hearing conference in June, 
1981 • Dean Nafziger from NWREL also attended a planning meeting in 
August 1980. Brief summaries of , meeting observations were provided to 
the NIE project officer for the NWREL evaluation contract. Plans 
developed by -the NIE in its position paper (Herndon 1980) and interim 
manoranda or documents also were data sources for the process 
evaluation. The NWREL NIE project officer coordinated with Enid Herndon, 
the MCT Clarification Process project officer, to assess consistency of 
interpretations* ♦ 

The evaluation team conducted interviews ■ with the NIE staff, team 

leaders and members and H. Paul Kelley who represented the hearing 

officer at the case development meetings. The initial interviews were 

conducted during January-February 1981. ^Subsequent interviews were held 

"... . • i . " 

with the-NIE-staff,_tto | .Team Leaders and Hearing Officers," i.e., Barbara 

Jordan and H. Paul Kfcll^y, following t the July hearing. 7 

Finally, summaries of the case development meetings, background 

documentation and papers on the Clarification Process were reviewed. 

Hearings ! 

Three primary data sources were used for obtaining information on the 
hearing conducted in Washington, D.C. July 8-10, 1981. First, the 
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questionnaire in Appendix A was distributed and participant responses 
collected during the third day of the hearing* Initial plans in the 
NWREL evaluation design were to colle'ct responses following each hearing 
day. However, the NIE suggested and NWREL agreed to limit' data 
collection at the hearing to the third day. The primary concern for 
reducing the data collection from each day to the third day was not to 
overly burden observers at the hearing. 

Three or four small group interviews of five to six individuals 
attending the hearing were conducted following each day of the hearing. 
Reactions to the effectiveness of the hearing and information gained at 
the hearing were discussed. Participants were encouraged to provide 
reactions to the process used to clarify the MCT issues. The interviews 
were unstructured and lasted for approximately 30 minutes. 

Finally, the evaluation teams 1 observations based on the three j 
hearing days were an additional source of data. The evaluation team 
attempted not only to observe the hearing process, but also noted the 
number of people attending the hearing and other audience reactions, 
e.g. , the degree to which participants stay&d for the entire day, or came 
and went during the day, or the extent to which some staff preferred to 

. observe the hearing on the TV monitors located in conference rooms 
outside the main hall. 
State Data Collection 

.„. The NWREL evaluation design included plans to collect data, from 



video-tape audiences within/states.. Data were collected in the following 

. i . 7 v 

states (dates are those for the data collection meetings). A brief 
description of MCT within^ the state is included. 



o California (October 28 and 30, 1981): Locally developed tests 

are used for high school graduation in response to a legislative 
mandate* ■ t 

o . Florida (November 2, 1981): A state-wide MCT is used for high 
school graduation based on a state \nandate. Graduation 
; sanctions were postponed due to litigation* 
o ! Illinois (November 6, 1981) : Districts can develop or use MCTs 
j as part of their assessments* There is no state mandate for MCT 
at this time, i.e., tests are not required as part of the 

assessment legislation. Many 'districts in Illinois have MCTs 

\* i . - 

and some use them for graduation. \ 
o Montana (November 6, 1981): MCT is under consideration. 

Competency based education programs is an issue in Montana and 

deliberations -as to the value of MCTs are underway. N < 

o Texas (November 4, 1981,): Similar to Illinois, Texas districts 

may use MCT tests for graduation or promotion, although the 

State Department of Education does not encourage districts to 

t 

use MCTs for graduation decisions. 

/ 

o Wisconsih (November 4, 1981) : Legislation is pending within 

Wisconsin on MCT. / A bill initially scheduled for hearing during 
the fall session is likely to be acted upon in the February , 
1982 session. 

The above states were selected based upon reviews by Gorth (1980) and 
Pipho (1980) and through interactions with the NIE and the Clarification 
Process team leaders. An attempt was nu\de to obtain variety in 
geographical representation, types of MCT programs and stage of MCT 



implementation across states. Stages of MCT implementation was important 
since the MCT concerns will likely vary as a function of whether there is 
a state test, loca^ tests and options, or whether MCT is still under 
consideration. Additional states that were contacted, but declined to 
participate were Georgia, Kansas, Maryland, Mississippi, New York and 
Virginia. All states except Maryland declined participation on the basis 
that the videotapes and materials from the Clarification Process would 

J! 

not be timely for the state. Maryland was interested in participating, 
but it was not possible to coordinate Maryland's schedule with NWHEL's 
data collection schedule. \ 

Each state was informed that a one-day session was planned in which 
the videotapes and written milerials from the Clarification Process would 
be reviewed. Each^state was asked to organize a group of approximately 
20 which would^rclude representatives from l£he state education agency,' 
i.e. /the chief state school officer or representative and other state' 
department staff , and LEA representatives including administrators, 
teachers and school board members. Additionally, it was requested that 
legislative representatives an^representativeia from special interest 
.groups also be invited. Appendix B is a copy of the letter sent to each 
state contact person and includes the agenda, for the one-day meeting and 
the type of participants desired] at the meeting. 

A questionnaire was developed and administered during the state data 
collections. Finally, as indicated in the agenda in 1 Appendix B, brief 
discussion sessions of ^proximately one-half hour were held in the state 
sessions following each of the one-hour videotapes. These discussions 
were tape recorded and transcribed for analysis. 

. < ■ \ 
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Special Interest Group Session , 

A session like those held within the states was heOjd in Washington, 
D.C. for representatives of special interest groves or national 
organizations on October 23,. 1981. The results in Tajble 1 reveal that of 
--th^^^sampl^of 19 groups contacted, 17 agreed to participate and eight 
actually attended the meeting on October 23. A national Title I 
conference on October 23 was expressed as the basis for the initial 
decline by the two groups* Last-minute schedule conflicts, other 
priorities and even rainy weather were cited as reasons for nonattendance 
by the others. „ . 



er|c 
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Table 1 

National Organizations for MCT Data Collection Meeting 

October 23, 1981 - ■ 



Organization 



Initially Interested Attended 



1. American Association of School 
Administrators 

2. American Federation of Teachers 

3. Association for Children with 
Learning Disabilities 

4. Association, for Siqpervision in 
Curriculum Development — 

5. Children's Defense Fund 

6. Council for Basic Education 

7. Council of Chief State School Officers 

8. Council for Exceptional Children 

9. Council for Great City Schools 

10. LULAC National Education Service Centers 



11. National Advisory Council on Education 
of Disadvantaged Children 

12. National Alliance of Black Educators 

13. National Association for the Advancement 
of Colored People ■ \ 

14. National Association of Secondary 
School Principals 

15. National Association of State Boards 
of Education 

16. National Coalition of Title I Parents 

17. National Conf erence of ,State Legislators 

18. ^ National Education Assoc iation . 

19. National School Boards Association 



/ TOTAL Yes 



Yes 
Yes 

Yes 

Yes 
Yes 
Yes 
Yes 
Yes 
Yes 
Yes 1 

No 
Yes 

Yes 

Yes > 

Yes • 

No 
Yes 
Yes £ 
Yes 

17 



Yes 
Yes 

No 

No 

Yes 

Yes 

Yes 

Yes 

No 

No 

No 
No 

Yes 

No 

No 
No 
No 
No 
Yes 

8 
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Thus, eight representatives of the spepial interest groins met and, 
provided input on October 23. School boards, administrators, teachers 
and special interest groups were represented, even though the sample was 
smaller than planned. Four of the participants on October 23 also 
attended one or more days of the hearing and were able to provide 
comments on their reactions to the videotapes in relation to the 
information presented at the hearing. - 
ER5/ENET. Meeting f " * 



The first day's edited -videotape was shown to a^grojjfe of ^ 8 



\ 



i^groj^p 

participants at the Evaluation Research Society /Evaluation Network 
meetings held in Austin, Texas, October 1-3, 1981. The group was 

composed primarily of individuals directly involved or concerned with 

' * v 

evaluations and represented multi-disciplinary backgrounds. ..Written and 

• - : ■ ■ 

tape-recorded reactions were obtained. 1, 
PBS-Dafca-Gol-lection- - 



The questionnaire contained- in Appendix C. was developed for the 
Southern Educational Communication Assbciatibn~(SECAr. "SECA distributed 
the questiorinaire to approximately 125 directors of instructional 
television and programming managers for PBS stations across the country. 
The major purpose for collecting data from these sources was to estimate 
the extent to which PBS stations broadcast the Clarification Process 
videotapes. Follow-up phone calls were made on December 2, 1981 to a . . 
random sample of 28 PBS stations to assess the representativeness of the 
limited resppnse to the written questionnaire from SECA. 

NWREL evaluation plans initially were to obtain PBS viewer responses 
from interactive systems such as the one at QUBE in Columbus, Ohio. 
Stations in Omaha, Nebraska and San Diego, California were ^contacted in 



addition to the QUBE station. The interactive systems were not in a 
sufficient number of households within either Omaha or San Diego to 
warrant collecting data* The QUBE station decided not to broadcast the 
Clarification Process series due to low interest and adequate other 
programming. " 
Analyses " r 

Data from the above sources were organized around the five evaluation 
questions. Table 2 relates the questions in the hearing and state data 
collection instruments. to the five major evaluation questions. 

Frequencies of response, means and standard deviations were 
calculated for the Likert type items and were analyzed across /and withih 
states. * / 

Open-ended responses obtained in the questionnaires and interviews; 
were analyzed to assess areas in. which there was consensus or agreement 
and~to~assess -the- variability am6ng^ndirViduals^n^tl^ir--assessment of 

the Clarification Process and its products. The objective was to obtain 

* J * ^ ' ' • _ • 

general trends from open-ended questions and interviews and to look for 

confirmation or discrepancies based upon Mthe object^^ve-^aantitative 

responses. Individual responses were noted, more frequent responses were 

■ i 

tallied and reported as such with interesting or contrasting responses j 
cited to provide/ an indication of the range of opinion. 
These analyses can be found in Chapter IV. 
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' Table 2 

Evaluation Questions Referenced to Questionnaire Items 



Evaluation 
Questions 




\ 1 

Questionnaire 
/ Iteiif/Iteras 


State/Specia: 
Interest Grouj 
Hearing Sessions - 





1, How appropriate were the The process of using testimony and cross-examination enhanced the. 
format and structure of the presentation of the issues* , ' 
Clarification Process in « 
presenting MCT issues, i.e., , The Clarification Process 1 use of direct and cross-examination 
in what ways did the , , enhanced the presentation of the issues. . . •'/ 

Clarification Process help 

or hinder the presenting The use of individuals 1 ' subjective judgments enhanced the 
of MCT issues? presentation of issues. I 

1 ' \ 

Thejlarification. Process' use of individuals' testimony and personal 
judgments enhanced the presentation of the MCT issues. 

The two teams presented a comprehensive case in support of their 
positions. 

a. pro team , - 

b. con team 

The use of individuals' subjective judgments enhanced the 
presentation of issues. '• ? 

Has there any important information that was omitted from the 
shows by either, the' pro or con teams? / 

Were the most; important MCT issues addressed in the hearings? 
What others, if any, might have been more important? 



H 

AO 



i' \ 



Do you 'feel that there are more important MCT issues that were not X 
addressed by the show? 

Overall, what do you think are the strengths of the hearings (shows)? X • X 

Overall, what do you think are the weaknesses of the hearings ^shows)? X X 



Table 2 (don't.) 



Evaluation 



Questionnaire 
Item/Items 



State/8peclal 
Interest Group 
Hearing Sessions 



,10 

' 0 



2, Did the information presented A variety of viewpoints m presented during the hearings (shows). X 
represent a fair diversity I 

of viewpoints on each iBBue The hearings provided a fair debate of the issues, X 
that was seen as clarifying 1 | 

and illuminating? , . Arguments were presented clearly byt X 

a. pro team 

' b. con team _ , , 

• - ' . ; ■; • • ■■■ >v . 

Has there any information' that you feel was not useful in 
clarifying theHCT issues? 

• • . •/ « ■ t . ■■ ■ . . •, 

3. Did the information presented Row would you rate your knowledge of HCT before the hearings (shows)? X 
add to the current under- j 

standing and knowledge of Did you gain any new knowledge from the hearings (shows)? X 
Minimum Competency Testing? . ..'*•" 

What, If any, was the most important new knowledge you gained 
from this show and how- will you use it? 

Has any information presented which was inconsistent with what 
you know about HCT? 

in your opinion what were the most persuasive points jade by i X 
a. pro team ' 

'. — brw 



Who were the most effective witnesses for 

a. the pro team and why? 

b, the con team and why? 



X 
X 
X 
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Table 2 (con't.) 



Evaluation 
Questions 



Questionnaire 
Item/Items 



interest Grouj 



4. Did the audiences perceive How would you rate your opinion of Hinimum Competency Testing (MT) x 

the information to be useful' before these hearings? i 
in terms of pending policy ' 1 

or program decisions? How would you rate your opinion of HOT after the hearings (shows)? X 

I What HCT issues are you most concerned about? 

» * ' ' 

Do you think the information in these shows will be useful 
to you in relation to these issues? How? 

Would you be interested in using the three one-hour edited tapes X 
from the hearings? < 

Rate the use of information from the hearings (shows) fort X 

a. formulating policy about adopting an MCT program 

b. assisting with implementing an HCT ) 

c. informing the general public about HCT 

d. other / please specify \ 



X 
X 
X 



Is the User's Guide useful as a supplement to the shows? 
Comment (s): 



What do you like about the User's Guide? 



How would you rate the User's Guide for 

a. formulating a policy about adopting an HCT program 

b. asslstingjithjhe Jmplementation=otan-HCT=program. 

c. Informing the general public about HCT 



X 
X 



Are there any changes that you would suggest that would make 
it more usable? ' 



3; 



3 
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Table 2 (con't.) 



Evaluation Questionnaire 
Questions [ Item/lteraa 



The shows ace about the same as a written evaluation, 
The shows • are not as good as a written evaluation, 

what other educational issues do you think could be covered 
effectively using the Clarification Process? 

Any general comment about this show? 



r 



5, What is the viability ofl thin How does the video presentation format compare to well written 
approach for other NIE I evaluation reports? 
efforts? ' The shows are better than a written evaluation. 



CHAPTER II I 
PROCESS EVALUATION RESULTS 

This chapter summarizes findings from the Phase It Process Evaluation 
Report (Bourexis, 1981). Complete process descriptions and evaluations 
with recommendations are contained in the Phase I Report. This 
discussion is organized around major stages in tho Clarification 
Process. The stages aret 

o Planning and Management Issues 

o Case Development Activities 
• * , o The Hearing , 

o Editing and Dissemination ; 
For each area and subtopics within the area brief summary comments 
are provided which give a description of the process and, where 
appropriate , evaluative comments based upoVi findings from the Phase I 
Report are made* 
Planning and Management 

Audiences* The audiences for the Clarification Process were 
identified by the NIE in the early planning stages and were reviewed in. 
the background statements in Chapter !• The target audiences clearly are 
diverse. NIE 1 s major audience was the set of policy or decision-makers 
within \tate and education agencies. The Clarif ication^Process was also 
intended to provide information which might be of interest or use to 
others including the general lay public. The primiary audience for the 
ultimate products of the Clarification Process taLS_£ wide ranging 
audience of local and state \staff. The audience at the* hearing was more 
representative of national organizations and federal agencies. It is 
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, possible that the representatives o£ national organizations could servo 
as proxios to the direct/ intondod audiences. The national organisations 
could have invited or arranged for actual members, e.g./ teachers/ 
administrators or school board members/ to be in attendance rather than 
relying on Central Office staff located in Washington D.Ci to attend the 
Hearings. An example is the 50 tickets each day which were allotted to 
the ^chool Boards and Chief State School Officers organizations. As it 
turned out/ it appears that the national representatives were ible to 
assist with dissemination through newsletters and other communications or 
efforts. 

The intended audience(s) affected deliberations on factors such as 
types of witnesses/ testimony and arguments to bo used in the 
Clarification Process. It is likely that the general/ wide audience 
specified for the Clarification Process contributed to uncertainty and 
conflicting directions in terms of focusing the materials and 
information. This uncertainty would have been minimized if clearer 
direction had been provided toward either a general audience or a more 
targeted audience of MCT "decision-makers^" Both groups would likely 
desire relatively nontechnical and sound data. However/ interests of the 
general public/ policy makers and implementers will likely vary in 
content or issues and in the level of information desired on any 
particular issue. 

Advisory group . An advisory panel representing various constituency 
grot^s was organized by the NIE in the initial planning stages. The 
advisory group which represented state legislators/ school board members/ 

state and local education staff/ provided the NIE with reactions an* 

" ° I ♦ " 

recommendations for the project. Its recommendations included procedural 



aspects such as reducing the trial like nature of the project, types of 
issues or areas tt;.be covered in the project and persons who might be 
potential team leaders or project participants. The NIE stressed this 
input and participation of the advisory panel as an important component 
of the Clarification Process. The advisory grog's influence on the 
process was minimal after the group was given an opportunity to react to 
proposed issues formulated by the pro and con teams. Advisory group * 
members were also invited to the hearing. 

K Based upon the modifications tb. the Clarification Process as a result 
of the advisory: grog's participation and input, it is clear that it was 
a significant factor in the Clarification Process. It also appears that 
its involvement contributed to broad knowledge of the project among 

i J •• 

.special interest groups or national organizations. This statement is 
based upon NWREL's telephone calls to set up interviews at the heai^ng. 
Most agencies contacted' were aware of the Clarification Process Hearing. 
It is in£>ortant to involve significant constituencies in the planning and 
formulation stages in any Clarification Process or evaluation. ' 

^Timelines . The timejJLiie-f or the portion of the Clarification Process 
primar-ily"~consi^ evaluation is the period beginning in 

October, 1980 with the training session for the teams through the 
dissemination of the Clarification Process materials as of October, 
1981. The Clarification Process actually was begun around November, 1979 
when considerations of the project and discussions with the advisory 
group A were initiated. Thus, .the time span from initial plans to 
completion was approximately two years; the time lapse between initial 
team formulations and hearings was approximately ten months. 

.... - - - . . - , * ». 1 
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It appears that the time span for the Clarification Process was 
somewhat less than that for a typical major evaluation or study and was 
similar to the timelines Jtat the Hawaii 3 on 2 adversarial evaluation 
, (Worthen and Rogers/ 1977) and slightly longer than Wolf * s application of 
the Judicial Evaluation Model at Indiana University {Wolf \ 1979) )\ It is - 
possible that the amount of time needed to plan and develop the project, 
i.e., November, 1979 to JUne-August , 1980 might be shortened in future 
applications as a result of information learned in this process. 
However, no clear time advantage appears' for the Clarification Process 
approach over other studies. Instead, the comprehensiveness of the study 
along with amount -of resources available will generally affect timelines 
and levels of effort more than whether a Clarification Process or 

.... ■ . ' t _ *. ' o 

adversarial approach rather than a traditional data collection. and 
analysis approach is used. If a study is a national effort which 
involves an advisory group in the planning and conduct, it is "likely to 
require a longer timeline than a state or local study not requiring the 
same level of coordination and input. 

Dissemination . As outlined in the Introduction Section, it was 

always the intent" /to produce videotapes from the Clarification Process 

■ \ ■ 

for dissemination. Additionally, it was intended that written materials /. 

would also be available. However, the specific agencies \or vehicles for 

disseminating these materials were not formalized until after February, 

1981. PBS broadcasts served as a primary dissemination vehicle for 



making the videotapes from the hearings\available. The Southern 
Education Communication Association (SECA^ was primarily responsible for 
publicity and facilitating dissemination i^" this area. Seve: 
compliments were made about the cjuality of the promotional ma€ 
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developed for the PBS shows by SECA. \ 
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The NIE sent information about the hearings to over 2,000 agencies 
during September-October , 1981. A more complete analysis and [evaluation 
of the dissemination efforts will be available through an int ernal NIE 
report in which major newspapers will be clipped to analyze the number 
and nature of articles on the 7 Clarification Process. Requests for 
materials will be docuanented and additional information as available will 
be included. It is probable that the dissemination plan will not. 
maximize the use of materials by the greatest . number of individuals. 
Although the potential market from the PBS broadcasts is certainly quite 
large, MCT is a relatively low priority and, will either result in 
nferginal broadcasts or broadcasts at "off times," such as Sunday 
afternoons during NFL broadcasts. From ours'contacts with PBS stations, 
it was clear that even though there was often personal interest among 
station staff or B&anagemerit, the PBS programming interests and priorities 
often did not allow for air time for the MCT broadcast. Finally, it is 
possible that NIE's internal evaluation of dissemination efforts which 
are outside the scope of this evaluation will yield a more positive 
picjture. This statement is based on the fact that the NIE evaluation 
will include the numbers of individuals requesting tapes, an updated, 
complete listing of PBS broadcasts and records of publicity on the 
Clarification Process. 

Clarification Process Roles and Team Selection v 

Team selection . Key roles in the Clarification Process included the 
Hearing Officer, team leaders and teams, the NIE and the contractor which 
assisted with the logistics, i.e. meeting arrangements, transcription of 
minutes, and handling other tasks as needed by the NIE. The NIE 
solicited recommendations from the advisory group for nominations of. the 



James Popham and the Heari 



Hearing Officer and team leaders. The advisory group^proposed names 
using criteria established by the NIE for these rol^fc, and NIE cojtacted 
potential-participants. 1— The resulting team leaders ., George Madausand — 



g Officer, Barbara Jordan, are clearly 



nationally recognized individuals. The key/steps in this process^ were 
the identification of criteria for the roles and the input provided from 



the advisory group. 



r 



Issues arose in/the planning stages related to maintaining equity 



./ 



between the teams^ Questions included how to deal with offers for 
assistance to teams from other 7 agencies. It was clear that thiis might 
result in inequity among resources available to teams. Eventual 



lly, it 



was decided to disallow ? wbernal contributions in either fiscal or other 



resources' unless offers were made equally to both teams. 

' . v ". ' / 

The team leaders were given wide discretion in selecting their 

/ . . ..." 

teams. The composition of the two teams differed on two dimensions. m 
First, the geographical proximity of the pro team members was much closer 



' / 



within 



than the- con team members. Pro team members were all located 
Calif qrnia, and the con team memtJers 1 locations included Colorado ; 
Illinois, New Jersey and Massachusetts. Second, the con team was staffed 
^ith a greater number of "senior level" technical/methodological members 
than the pro team. ' \ 

The geographical distance between members made it more difficult for 

-" ■ c i 

the con team to interact in their case development. Also, it appeared 



that the types of activities that individual team members participated in 
differed between the pro and con teams, either as a function of Athe team 
leader's orientation/style or the constitution of the teams.' \ 
Specifically, data collection, analyses and case building appeared to be 



more evenly distributed? among con team members than pro team members , 
e.g. , most of the interviewing by pro team members was done by two staffs 

whereas* con team members shared interviewing responsibilities more 

• • . • ■ ■ • . <* ■ ' 

equally* < 

It is not possible in this case to separate the effects of team 

structures from the effects of arguing pro pr con issues and their 

implications on the types of cases or' presentations developed* However, . 

it is clear that logistical constraints were greater for the con team 

than the pro team. The potential benefit for this constraint was greater 

. .... . ' - . • 

representation within the team of differing groups and higher level 

technical expertise. It is equally clear that advancing a pro or con 

argument affects the type of case which can be built , e.g.,, the coi^team 

encountered greater difficulty in obtaining witnesses filling to testify 

against their MCT programs than did the pro team in obtaining witnesses 

to testify for /their programs. ' 

Hearincr Officer. Th^ Hearing Officer, Barbara Jordan, was assisted 

tented-all case development meetings. * H. Paul 
Kelley served as a moderator and facilitator during case development j 
sessions and occasionally ruled on questions related to procedure, 
presentation of evidence or groundrules. It was clear from team 
interviews and comments from people attending the hearing, that Barbara r 
Jordan's background added credibility to the Clarification Process. 
Evidence was also provided that H. Paul. Kelley and Barbara Jordan jointly 
performed a function which was needed in the Clarification Process. 

NIE's role . The role of the NIE after the planning stages of the 
Clarification Process was intended to be a decentralized and nonobtrusive 
role. It was stated at the outset and during the process that the NIE 
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wanted the team leaders and hearing officers to manage the project 
without strong direction or guidance by the NIE. This objective was met 
with varying degrees of success, during the tf case development stages. On 
seme occasions, it was expressed that the NIK should simply state what it 
is that they wanted as in the area of issues to be addressed. On other 
occasions , it was stated that NIE •' s decision or rwe^ 
particular data to be presented at the hearings was not appreciated and 
was viewed as intrusive or "heavy-handed. " > - , 

The role of the NIE or other project sponsors in the Clarification 
Process is likely to always be a key one. The role would be facilitated 
by limiting the sponsor's interaction during the process to only the 
Hearing Officer. However, it is difficult to perceive a sponsor in 

C\ ..„•'.•••■ 

. . , - ■ * . ... ... .-. & 

attendance at case^tevelopment meetings not interacting on its behalf on 
at . least seme occasions. These interactions will probably be received^, 
with mixed acceptance. ^ Thus, limiting the sponsor's interaction to 
Include only the Hearing Officer, might also affect the way the sponsor 
would monitor the project. The more important factor is to be clear on 
the individual roles and responsibilities rather than to maintain any 
specific Set of rules across all applications. This role could be 
altered in other Clarification Process applications or adaptations. A 
sponsor could leave the project totally in the hands of the Hearing 
Officer in future applications. This obviously will result in the 

sponsor having less control and influence. However, if an objective is 

'. ■ ' 

that the process should be able to "run itself," this approach might be 
warranted. 
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Contractor role / The contractor performed a major role in the 
Clarification Process • In addition to handling logistics and 
arrangements for the NIE and jteams, the contract^jwas needed and 
responsible for subcontracting the pro and con teams' contracts. As is 
elaborated upon] in the Phase I report and later yf'tlV Recommendations 
Section/ the role of the contractor in the Clarification Process could be 
assumed by either the Hearing Officer or the project sponsor in future 
applications. This is particularly true if the contracting/grant process 
is different than that for the Clarification Process in which another y 
agency was required to let, and manage the contracts to the Teams and 
Hearing Officer, ^f the logistical arrangements can be managed by the 
Hearing Officer or sponsoring agency/ greater efficiency wo^As^result 
from fewer roles to coordinate and simpler communications. An example in 
which the contractor and Hearing Of f icer: role w^s held by the same 
agency/individual was the Hawaii 3 on 2 evaluation (Wort hen and Rogers, 
1977). ; , \ 
Training for Teams , / 

An initial training session was qonducted for \^he^teams in 
Washington D.C. on October 9-10, 1980. This session was intended to ^ 
acquaint the teams and Hearing Officer with the background to the 
Clarification Process and ihe Judicial Evaluation Model from which the / 
Clarification Process had been adapted and to initiate the clarification 
Process case development. Questions Which were raised at the training 



/ 
/ 



session ai?out the Process included: 

7 ' / - " ' ■ . ■ 

/• How do we maintain the clarification intent of the process while 
still arguing pro and con? ' 



/.-. ... 

/• • •■ 

/ • 
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• This approach is limited to only twcv sides for each issue. * 
Since there may x be more than two positions on any gLver^ao^ie, 
how can the process serve to bring out the different, positions?, 

• To what extent must each team involve relevant constituency 
groins?- 

• How should Minimum Competency Testing be defined? 

• How should HOT issues be framed? 

+ How many individuals /from each team can be case presenters? 

• Which team will present its case first? 

• How many witnesses can be called by each team to testify? 

• What standardized procedures, if any, should be used in 
interviewing witnesses? \^ ? 

• What is the role of the Hearing Officer? 

• Should challenges to witnesses be allowed during the hearings? 
These were discussed at the session with most being resolved during 

the course of: the Clarification Process. It appeared that questions 
related to groundrules, rules of evidence and procedure were fairly clear 
following the training session, whereas, approaches or decisions for 
framing the issues were less clear. The importance of arid difficulty in 
framing issues within "the Clarification Process is similar to problems 
described by Owens and Hiscox (1977) and Wolf (1980) in which they 
outline issues formulation as one of the ipajor concerns in adversarial 
evaluations* 

Finally, it appears that additional practice or rehearsing in the 
areas of direct and cross-examination would help the process. Based uppn 
comments made by hearing observers and others it was perceived that 
direct and crosi3-examinatipn was stronger following the first day of the 



hearings. Most likely this was more a function of the experience and/ or 
confidence gained during the first day and less a function of the types 
of witnesses or testimoi^ bet days. Also , no evidence was cited 

by interviewees that the attorneys on the con team were more or less 
effective in direct and cross-examination. 
Issue Selection and Procedural Rules 

Issue selection and framing was a\major step in~the Clarification 
Process. Issues and procedural rules, were to be adopted by December, 
1980. The Clarification Process used one phase to identify issues , 
whereas other judicial evaluation approaches divide issues generation and 
issue selection into separate stages. The work of the NIE in previous 
MCT contracts and the input from the advisory committee assisted with 
identifying many of the issues. The issues which w6re finally selected 
addressed, effects of MCT on the students, curriculum and public 
perceptions and are detailed in the introduction section of this report. 
Much of the training session in October, 1981 and the two meetings 
" following were used to review ana finalize the issues. It appears issues 
formulation is still the step most difficult to accotaplish. Poptiam 
(1981)\nd Madaus (1981) outlined alternative approaches to framing 
issues. This might make issue formulation more efficient in subsequent 
applications. \ Some port ion of the time devoted to selecting and framing 
the issues would be more profitably used in case development and review 
stages. These statements are supported by direct observations of jthe . 
evaluators, interviews with team leaders and the Hearing Officer. The 
concern with formulating issues in the Clarification Process* reinforces 
the observations of Worthern and Rogers (1977), Owens and Hiscox (1977) 
and Wolf (1980) as to the importance and effort which must be given to 
this step in adversarial type approaches. 



Much- of this effort' was directed toward attempts to insure that cases 
•" / \ . ■ 

would be built around common issuek and arguments. It is quite possible 

that arguing a pro or con side automatically assures; thai: different 

factors will be stressed. The process Should singly allow for the 

probability that pro and con cases will ria^ argue perfectly parallel 

\ 

subissues. Noi£>arallelism in cases should riot to be viewed as a weakness 

. ■ ' j ' V '• -V:" . " . ' \ / 

but accepted as a characteristic requiring attrition. \^ 

% A paper by Wolf (1980) was used in adopting Woundrules for the 
Clarification Process. The groundrules included floras for collecting 
witness testimony and documentary evidence. These forms facilitated 
sharing information among teams. The rules of procedure included 
guidelines for opening and closing remarks, examinations of witnesses and 
making objections. Rules of evidence included relevance of information, 
depositions, Use of hearsay, documentary or opinions as evidence. 
Although sever a l\re visions were made to the rules, they yere only minor 
revisions. Thus,\the formulation and application of the groundrules, 
rules of procedure and rules of evidence ran quite smoothly in comparison 
/to the process of generating' or wording issues. . 
Additional Case Development 

Complete minutes and summaries of each case development meeting were 
compiled. These are available from the NIE for anyone who might be 
interested in adapting the Clarification Process to their study. A phase 
following and partially overlapping with the final selection of issues 



was the development/of stipulated agreements that prohibited debate or 

discussion of certain topics. - These were primarily concerned with- using 

MCT (a) at each grade for certification/classification, (b) for teacher 

evaluation or (c) for resource allocation. Discussions about the degree 

« 
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to which stipulated agreements should be presented at the hearing were 
held. Tlje final decision was to review briefly the stipulated agreements 
at the beginning of the hearings. . Based on comments by hearing 
participants and eyaluator's observations , future applications should r 
elaborate more on stipulated agreements at the hearing and in the ' 
videotapes. Providing the audience with pertinent information that led 
the teams to stipulated agreements — instead of simply stating they exist, 
because the teams agreed that MCT was inappropriate for those 
purposes—would Help the audience understand the function of the 

/ * c 

stipulated agreements. The process would be stronger if the rationale 
for the stipulated agreements was provided. One possible variation for 
introducing the stipulated agreements , is^o address them in testimony as 
was done with handicapped students. The pro team agreed with the con 
team witnesses' testimony against using MCT with handicapped : . ,.^i:*vsv 

Major case development activities were to occur in meetings from 
Pecember, 1980 through June, 1981 just prior to the hearing. These 
meetings were generally to include presentation and discussion of each 
team's position statements, review of case development and data 
collection plans, arid discussion of direct and cross-examination plans. 

Meetings were also used to discuss logistics for the hearings. Logistics 

| f 
involved order of team presentation, amount of time to be devoted to 

direct and cross-examination and the number of issues to be addressed 

each day. Factors which influenced these discussions include^ tha fact 

* ■ 
that witnesses could not attend^ all three days, nor could the budgets 

*■ " .,' 

support the, per diem of witnessed for three days. 



Activities related to the PBS videotaping of the hearings for later 
editing and , broadcasting on PBS stations presented a major logistical 
factor. These JLogistics ranged from type of clothing which might be 
needed to types of witnesses or testimony v which might be effective. ,Team 
leaders and members expressed their perceptions that the national 
broadcast of the edited videotapes of the hearings affected their plans. 
These effects included types of witnesses they might select and types of 
'cases they might want to argue. 

• -7 ■ . * ■ ' ■ 

fi it is difficult to estimate, the amount of the case development time 

ifhich was devoted to logistics and the television broadcast. However , it 

f ' ' ' < •' ' • ' ' 

'■'"/iss clear that substantial portions of meeting time,' were devoted to 

inactivities other than those directly related to the substance of the 

leases and hearings. This is consistent with experience in other 

/ \ . • - 

•^applications of the adversarial approach in which Popham (1977) states , 
that of three days designed to exchange cases, two were devoted to 
procedural matters. Future efforts should increase or maximize the 
proportion of time directed to addressing the teams 1 cases. 
, The case development steps or phases were maintained throughout the 
Clarification Process although timelines were not strictly followed. In 
the Februaery, 1981 meeting both teams provided their basic arguments, 
outlined types of witnesses they jintended to call and discussed potential 
data collection plans. Although questions were raised at the meeting 
about the degree to which the arguments might be responsive to each 
other, little discussion ensued. Relatively more attention was given to 
the PBS Broadcast (note this was the^ first meeting in whi-ch PBS was 
involved). — - 
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The April, 1981 meeting iiK:luded>^scussions on the witnesses to 
testify for each teamj and types of arguments or cases to be built. $ A 
major point of discussion at the meeting was whether some pro team sites 

fell withi/i the MCT definitions. Again, this experience was similar to 

° • * 

the differences in interpreting definitions or ru£fes of evidence found in 

other application of adversarial approaches (Popham and Carlson, 1977; N 

Worthen and Rogers, 1977). In April the teams expressed strong concerns 

about their ability to vuay within the timelines for developing their 

cases. Given this need, the May 22 meeting was postponed to June 12. 

3 The issue of relating the cases to one another was also raised at the 
April meeting in which at least one-third of the meeting was devoted to 
reviewing PBS related plans and the NWREL evaluation plans. 

The June 12 meeting was the first in which delays- in the case 

.development progress appeared to significantly affect the process. The 
pro team had not received the con team'^s summary of witness testimony on 

direct examination, and thus had not outlined their cross-examination as 
* ■' * 

was planned. The con team felt that the pro team witness summaries were 
*\ ' V ■ . * 

not sufficiently detailed , for them to prepare summaries of their 

cross-examination. It appeared that the amount of work that needed to be 

i 

done within the time limits caused much of this concern. It also 
appeared that unresolved issues from earlier meetings, e.g., were the 
teams' cases responsive to each other, affected decisions and ability to 
resolve concerns in the June meetings. As the hearings approached, the 
"case"dwel^me^ This 
was in part a natural trend which is a function of work distribution in 
any major effort; it is also likely that the newness of the Clarification 
Process and the PBS television aspects contributed to these stresses. 
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Future endeavors might reduce these stresses by decreasing the proportion 
of 'case development time which is taken for logistics or factors such as, 

4 

in this instance, tjhe television broadcast and to increase the time spent 
* ' * ■ i 1 

on cases, i.e., building direct and cross-examination plans. 

• \ 

Rules of procedure and evidence and ground rules were fairly well 
accepted at the outset of the Clarification Process. However, tensions 
arose during the case development th*t are similar to those reported in 
other adversarial type processes (Pop ham and Carlson, 1977; Worthen and 
Rogers, 1977). Tension of this type is to be expected, and knowing or 
recognizing it should reduce the senstiveness of individual feelings and, 
general negative influence on the process. 
Hearing 

Arrangements. Detailed plans and arrangements were directed by the 
NIE Project Officer, Enid Herndon, with the assistance of the McLeod 
Corporation. The previous stages in the Clarification Process 
facilitated preparations for the hearing. For example, logistics such as 
amount of time to be devoted to direct and cross-examination within each 
day were agreed upon; timekeeping procedures were established to monitor 

amount of time used by pro and con teams on direct and cross- 

\ ■ 
examinations; an agenda outlining the order for witnesses was developed; 

and plans to have the Hearing Officer review the definitions of MCT, the 

stipulated agreements and agenda for each hearing day were outlined. 

However, the effort needed to adequately plan for the hearing^should not' 

be underestimated. As the Clarification Process neared the^iearing 

stage, it required the full time attention of the NIE Project Officer 

witl| substantial assistance from others "such as the McLeod Corporation. 

The documentation available from the NIE on the plans and logistics will 



facilitate future applications. Finally , the logistics at the hearing 
went smoothly. This is attributable to the detailed plans and 
arrangements .made prior to the hearing. 

Audience . » The. location of and audience for- the hearing are factors 
which deserve comments. The decision was made to conduct the hearing in 
Washington/ D.C. and to extend 500 invitations for each day* Although 
the videotapes could provide information to a large audience, it was 
clear that the NIE and teams agreed that the- hearing itself was an 
important opportunity to provide information to the audience. The 
audience described in the Phase I report was invited primarily through 
contacts with national organizations located in and around Washington^ 
D.C. The invitations were disbursed in early June, 1981, approximately 
one month prior to the hearings. The number of observers at the hearing 
ranged from approximately' 140 in the- mid-morning of day one to fewer than 
* 60 at the conclusion of day three. Thus, the number in attendance was 
substantially less than the anticipated 500 for each day. This might be 
attributed to several factors. First, the date on which invitations were 
sent might have been too close to the actual hearings to allow people to 
arrange schedules and secure support* to attend the hearing. Second, 
individuals at the hearing also indicated that it was difficult to attend 
all three days given other demands on. their schedules. This response was 
also encountered when people were contacted prior to the hearing to 
determine whether they would be interested and available '-'for interviews. 
Several indicated that they planned to attend one or more days but would 
find it difficult to attend all three days. Third, the location of the. 
hearing, while convenient to federal and' national organizations, did not 



minimize the distance which would be required for state or local j 
representatives to attend. The NIE stated early in the process that the 
federal agencies were not a primary audience. It was estimated that 
approximately 20% to 40% of the audience at the hearing were federal 
agency representatives. 

If one assumes that the full information presented during live 
testimony at the hearings is valuable* it follows that the live audience ^ 
at the hearing is an important grot^>. Thus, greater attention might be 
given in future applications to ensuring wider participation by the 
primary audience. Recommendations to facilitate this are provided in the 
Phase I Report and will be reviewed in the latter sections of this report. 

Hearing Observations . Two aspects of the hearing were highlighted as 
important to the Clarification Process. First, the degree to which the 
definitions of terms and stipulated agreements were reviewed was felt to 
be important by those participating in interviews by the evaluation 
team. Specifically, those interviewed felt that insufficient elaboration 
was given to explain clearly the definitions and stipulated agreements. 
The nature of some of the stipulated agreements appeared to v confuse the 
audience. A prime example is the stipulated agreement that barred debate 
about using MCT at each grade level for promotion/retention. It was 
unclear to a large proportion of individuals interviewed as to why the 
teams then continued to debate the use of MGT as a requirement for 
graduation or promotion/retention. Why the teams agreed on not using MCT 
at each grade level but disagreed on using MCTs at some grade levels was 
confusing. Second, several participants felt that the cross-examination 
was not effective in engaging the arguments across pro and con teams. 
Interviewees suggested that the cross-examination by the pro team was 



more effective during the second day than during the first. Although 
several participants felt that the reason for the "ineffective 
cross-examination" was the fact that the team* leaders were not attorneys 
or experienced in cross-examination, no 'one suggested that the two 
attorneys on the con team who engaged in direct and cross-examination 
were more or less effective than the non-attorneys. Practice or a dry 

run prior to the hearing would facilitate this area in future efforts. 

\ 

It is possible that attendance at the hearing by the primary audience 
should not be a goal. The complete tapes and transcripts were made 
available to the public. The only advantages, therefore, to attending 
the hearing were the immediacy of the information and the "live" effect. 
The "live" effect seemed to be initially interesting, but the attendance 
dwindled as the hearing, progressed. After the first day, most NIE staff 
members preferred to observe the proceedings from a television monitor in 
a conference room adjacent to the auditorium. When asked about this, the 
staff replied that it was easifer.to see facial expressions and pick up 
innuendoes in testimony from the monitor. It was also easier to interact 
and discuss witnesses' testimonies in the television monitor room than in 
the auditorium. 

Editing and Diss em± nation of the Clarification Process Materials 
It was decided that the pro and con teams would have primary 
responsibility fear editing the full transcripts and videotapes of the 
hearings to produce the three one-hour tapes. This decision was 
clarified during the February, 1981 meeting\n Washington, D.C. at which 
the PBS staff members were initially present. \lt was clear from the 



outset that the team leaders and teams would be iWolved in editing, 

.... • \ ■ . ■ " \ " " 

reviewing or approving the edited tapes. Apparent ly\t he team leaders 



were not clear on the extent to which they would have' primary 
responsibility for the editing and the amount of effort which this would 
require. Editing is a key step in the process since the edited tapes are 
likely to be used much more than the full tapes or transcripts. An 
aspect of the editing which affected the hearing ^nd Clarification 
Process was that it . was not possible within time and budget to edit 
across hearing days to produce tapes that would address each issue. It 
was possible, however, for PBS in its technical editing to reorder some 
witnesses' testimony so that the proximity of pro and con- team arguments 
was closer than it was in the hearing. This aspect of the editing was 
represented to favorably by those who 7 viewed the full hearings and the 
edited tapes^^oTC^rn about whether or not the edited tapes' repeated 
fairly the entire hearings and testimony was expressed at the hearing and 
in the showings of the edited tapes. Thus, strong importance is felt by 
the audiences toward the editing of the tapes. 

Comments from viewers about the editing were generally favorable with 
only minor specific reservations. Samples of unfavorable reactions 
concerned Mike Farrell as he introduced the NEA representative from 
Wisconsin by describing thk NEA as the largest teacher's organization 
within Wisconsin. Same viewers felt that this was a gratuitious 
advertisement for the NEA and might imply a partial attitude toward that 
teacher organization. Again, unfavorable comments were infrequent and 
were riot widely shared. — 

Finally, it is critical that the participants have a strong 
.involvement in the editing as was done here. Placing the pro and con . 
witnesses together was viewed as- an advantage of the videotapes to the 
actual hearings. 



Process Findings 

The above discussions were intended to provide a summary of the 
Phase I findings in which major stages of the Clarification Process were 
described and evaluative comments were provided. Conductions and 
recommendations which might be helpful in future applications will.be 
summarized in Chapter IV of this report. While it will not always be 
desirable to clearly separate recommendations based on the process and 
outcome evaluations of the Clarification Process,, it is anticipated that 
conclusions and recommendations will facilitate conducting future 
Clarification Processes and will ensure that the outcomes are more useful 
to the intended audiences. '• 




CHAPTER IV 
INFORMATION PROM THE AUDIENCES 

This chapter contains summaries of audience reactions to the actual 
hearing or the edited tapes, s The information for, this chapter came from 
the variety of sources mentioned in Chapter II. The three sources t 
reviewed in this section are ^ the heafing audiences, the audiences that 
viewed the edited tapes and the PBS progranraiing managers. These sources 
will first be briefly described in this fchapter. Following these 
descriptions, the reinainder of the section is organized around the five 
major questions and their subqiiestions as listed in Tabie 2 at the end of 
Chapter II. ^ 
Description of the Hearing Audience 

On July 8, 9 and 10, the MCT Clarification Process? hearing was held 
as planned. It was anticipated that 500. persons per day might attend. 
Tickets were distributed on the basis of this limitation and procedures 
were set that would allow only those persons with tickets to attend. 

The actual attendance at the hearing was much less than had been 

expected. Audience counts were taken at different times during the three 

i 

days and the following table represents those counts. V 
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» Table 3 

Approximate Attendance During Hearings 



July 8 


110 






135 


i n • h n 


140 




115 




115 


2:20 


115 


4:15 


Q6 


5:00 


69 


5:40 


60 


,T n 1 v Q 




q . n r 


74 


±u : ±d 


115 


i. • O \J 


98 


A m (\(\ 




c ,nn 

D SUU . 




5:30 


39 


J^uly 10 




9:10 


57 


10:50 


64 


11:45 


60 


2:15 


59 


3:10 


54 


3:45 


42 


4:15 


46 



Beginning of pro witnesses 
After recess 



After recess 



After recess 
Final witness 



The questionnaire that appears in Appendix A was distribute^ to the 
audience in the morning, July 10, and throughout the day. In addition, 
persons were stationed at the exits, to remind those persons leaving early 



to return the questionnaire. Approximately 80 questionnaires were 
distributed and 62 were returned. / 1 

The questionnaire contained N^ome preliminary demographic data to 
ascertain the persons* agencies and their roles within those agencies. , 
These descriptions appear in Table 4. Totals less than 62 resulted from 
some individuals not responding to all items. 
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Table 4 

Agencies and Holes Represented in Hearings Audience 



1 Agency represented) 


N 
a 


a. 


Federal , 


14 


(23%) 


State 


13 


(21%) * 


Local 


6 


(10%) 


National Organization . 


1 1 

XJ 




Other 


15 


(24%) 


Total 


61 




Missing 


1 


(.2%) 


Roles 






Responsible for policies about; 


8 


(13%) 


whether an MCT program 






should be implemented 






Responsible for the 


15 


(24%) 


implementation of an 






JMCT program 






Directly affected by an MCT 


10 


(16%) 


^ program 






Other 


25 


(40%) 


Total 


58 




Missing 


, 4 


( 6%) 


The agencies represented were spread evenly across 


four of the five 



categories, with the fifth, Local, being only 10 cpercent of the 
audience. The distribution of the actual audience differs substantially 



from the intended audience which was hoped to consist of mostly state and 

\ 

.local personnel'. 

For roles represented by the audience, a little over one-half (53%,) 
represented the intended audience for the Clarification Process. It is 
interesting to note that the role of implementing an MCT program was the 

most represented among the intended audience. In later questions, the , 

< * ■ 

usefulness of the MCT Clarification Process for implementing MCT programs 
received one of the lower ratings from the audiences. 



Description of the Site Visit Audiences 

Appendix B contains the letter that was sent to the different State 
Departments participating in the site visits. Included in \it was a list 
of roles and types of persons that should attend the meeting. State 
Department personnel were very cooperative in attempting to obtain these 
representations. When the desired audiences were not obtained; reasons 
given were mainly limited money for travel^ and limited time to devote to 
a full day meeting. Each state contact person made an effort to get a , 
legislative aide or representative to attend, but only the, Wisconsin 
meeting included state legislature representatives. Other priorities, 
i.e., legislation, budget considerations, were cited as reasons j 
legislative staff did not attend in other states. Each group consisted 
primarily of state or local staff directly interested in MCT. More - 
administrators were represented than testing personnel. One or two 
university affiliated persons were also in attendance at most sites. 
Exact counts of the audience's roles or af f iliationfewere not recorded. s 

Three of the states had fairly large participation. Two of these, 
Montana and Wisconsin, are states where MCT is either being considered 
(Montana) or a decision is forthcoming (Wisconsin) • In the third, 
California, two meetings were arranged with one. in Los Angeles and the 
other in San Francisco. This reduced the travel for some of the 
participants and probably was a key factor in their attending^ 

One of the first questions that the evaluation was concerned with was 

» • \ . • / 

whether the participants at the sites had seen the documentary or the 
edited tapes before the meetings. Table 5 contains the information 
concerning this question. It is clear that very few had seen~the shows 
by the time of our meetings in late October and early November. 
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Table 5 

Participants Who Have Previously Seen the Shows 



Having seen documentary: 
No 
Yes 



Having seen shows: 



Show 1 



Show 2 



Show 3 



/ 



N(%) 
131 
6 



\ 



No 
N (%) 
132 (98) 

134 (99) 

121 (98) 



(96) 
(4) 



1 from Illinois 
4 from Wisconsin 



Yes 
N (%) 



3 (2) 1 from Illinois 
2 from Wisconsin 

c ■ 

2(1) 1 from San Francisco 
1 from Illinois 

2 (5) 1 from Illinois 
1 from Wisconsin 



The PBS Stations 

The data collected and analyzed from the PBS programming managers 
were from questionnaires that were distributed by SECA. There was a 
concern about how representative these responses were of all the PBS 
stations. A jcandom sample of 28 stations from the Directory of 
Information Sources for Public Television were contacted in December and 
were asked if they had shown or planned on showing the documentary or the 
edited tapes. Fifty percent of the sample called responded positively. 
This compares to 68% (19 of 28) of the sample that returned 
questionnaires. Given a standard error of approximately 9%/ this implies 
that the responses to the questionnaire over represents the stations that 
will air the programs. Comments in the evaluation report from the PBS 
survey should be interpreted with that fact in mind. 
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X. How appropriate were the format and structure of the Clarification 
Process in presenting MCT iaauea? 



Question la: Did the use of direct examination (testimony) and 
cross-examination enhance the presentation^ of the issues? 
Figure 1 represents the responses to this question ,that appeared on 

both the questionnaire at the hearing and the state visits. Table 1A in 

Appendix E contains response summaries by state. 



100% n 
90% 
80% 
70% 
60%- 
50%- 
40% 
30% 
20% 
10%-) 



Hearing 
N=62 




Sites 
N=120 



1-2-1 




Strongly Disagree Agree Strongly 
Disagree \ Agree 



Strongly Disagree Agree Strongly 
Disagree Agree 



Figure 1. Audience responses to The Use of Direct and Cross-Examination Enhanced 
the Presentation of the Issues. (Responses shown in percents.) 
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Audiences agreed that testimony and cross-examination enhanced the 
presentation of the issues. Eighty-two percent of the audience at the 

V.', .' 

last day of the hearing anc 83% of the site-visit audiences agreed to 
some degree with the statement. Of the state visits, the most negative 
reaction came from California where 31% (10 of 32) disagreed /with the 
statement. In FloJfcfcga and ; Illinois, there was no disagreement with the 

statement. \ \ 

Audience interpretation of •"enhanced" should be considered* In 
California, there were several negative comments in the open-ended 
responses about the use of emotional data and the lack of factual or 
back-up data. This could account for the negative response to the 
question. On the other hand, in Florida, which had no negative 
responses, there were remarks about the discomfort with "bleeding heart" 
testimony and the concern with misrepresentation of facts. The v 
relatively high ratings in Fig* ^ 1 combined withiindividuals 1 comments 
imply that the word "enhanced" va» probably interpreted as adding to 
interest level, but not necessarily providing valid data. 

Finally, comments at the hearing and in state visits about the 
effectiveness of cross-examination and to a lesser extent the direct 
examination suggested that simply presenting evidence without the 
adversarial connotation might be effective. That is, the adversarial 

a 

nature of direct with cross-examination appeared to enhance the interest 
value but not necessarily "sharpen" the information produced from the 
Clarification Process. 
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i Quaation lbt Does the use of individuala 1 personal judgments enhance 
the presentation of the MCT issues? 

The Clarification Prooeas offered "the actuation where a witneaa wao 

it 

allowed to expand on his/her own beliefs and was not necessarily required 
to produce hard data to back-up statements. At the same time, many 
evaluation procedures fail to gather this subjective data that can 

4 

provide important additional information. The question of whether the 
use of personal judgments enhances the presentation of issues was asked 
at the hearing and after showing all three tapes at the state visits. 
The data collected are summarized in Figure 2. Table IB in Appendix E 
contains summaries by state. i{ 



Hearing 
N-68 




Sites 
N=v121 



1 — 



12 




-Strongly Disagree Agree Strongly 
Disagree A{ree 
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Figure 2. Audience responses to The use of Individuals' Testimony and Personal 
Judgments Enhanced the Presentation of the MCT Issues. {Responses shown in.percents.) 
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Tho data collected Horo are vary similar to the data mentioned 
earlier concerning the use of direct and crone-examination. Th^audi'ence 
appears to 'have equated the uoo of personal judgments with the use of 
direst and cross-examination in the hearing. Again, comments support 
that the personal testimony contributes to the interest value but not 
necessarily to the quality of information gained from the Clarification 

♦ 

Procass. Although the adversarial process of direct and 
cross-examination is intended to bring into clearer focus the points 
under discussion, little evidence was found that this occurred in the 
Clarification Process, even though viewers agreed that personal judgments 
and examination jonhanced the presentation of information. 

As will be elaborated later, the criterion used to evaluate the 

i ! . 

usefulness of the Clarification Process greatly affects h^w these 
comments are interpreted and the subsequent utility of the Clarification 
Process. We proposed a criterion that information .rated as useful by the 
audiences could be a criterion for utility. In> this sense, the aspects 
of personal testimony and direct/cross-examination were seen as 
strengths; strengths not necessarily proposed for the adversarial 
approach, as described by Wolf (1979). "In addition to interest, a^ 
perceived benefit of using personal testimony within an adversarial 
format is that individuals' "biases" are more clearly discernible than 
might otherwise be possible. Highlighting ispues and variety of persons 
were also cited most frequently as strengths of the Clarification 
Process. See Question If. , ** , 
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Question lc: Did the teams present comprehensive case^ in support of 
their positions? 

.The question of the comprehensiveness of each team's argluments yras \ -j 
asked at the hearing and after all three tapes had been shown at the 
state visits. The data from the ^est ion are summarized in Figure 3 for 
the hearing and Figure 4 for the state site visits.' Table 1C and 
Appendix E contains responses by states. ; - 
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Figure 3. Hearing audiences' responses to Comprehensive Cases in Support of their 
Positions Were Presented by the Teams. (Responses shown in percent s.) 
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Figure 4. Site audiences' responses to Comprehensive Cases in Support of their 
Positions Were Presented by the reams._(Respcwises shown in percents.) 



It appears from the dati in Figures 3 and 4 /that the audiences saw 
the pro team as presenting a more comprehensive case than the con team. 
This was. true throughout the site visits except for Montana where neither 
, team was rated superior to the other.. Finally, both teams and, thus, _ the 
Clarification Process were rated as providing comprehensive cases. 
Comments support that the comprehensiveness is , largely felt to result 
from the variety of witnesses, MCT programs and testimony/evidence which 
were ^presented. No strong consensus existed as to why the pro .team w^s 
felt to present a more comprehensive casa t>an the con taam. Comments 
included that the con team's case was focused more on technical issues 
related to setting cut scores or unreliability of tests. It also is our 
opinion that this was partly a function o£ the ease with which the pro 
team's arguments could be identified with the issues under debate. The 
pro' team's case was generally clctfevly related to effects on students, 
curriculum or perceptions. \ 
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Question Id: Was there any Important information that was omitted - 
from the shows by either of the teams? ^ - 

After all the shows had been viewed at the site visits, the audiences 
were asked to identify any important information' that they^felt had been 
omitted from the shows. The list below represents the response to this 
question* : 

Explanation of variety of MCT programs and laws (7) 
Implications to possible teacher evaluation (3) 
Political issues (2.) 
Local/state control issues (2) 
More definitions and explanations (2) 
Costs of MCT programs (2) 

MCT misrepresented as sole-criterion (2) * 
MCT is only a part of competency-based education (2) 
Criterion-referenced testing role (1) 
Setting cut scores (1) 

Abilities necessary for local construction of MCTs (1) 
Clarify differential standards (1) 
Special group of students (1) 
Technical aspects of a good MCT (1) 

Need for procedural* modifications from year to year' (1) 
Separating testing controversy from MCT issues (1) 
Pact that MCTs are minimum (1) 
Court actions (1) 

Team members and witnesses' current roles as consultants (1) 
MCT recordkeeping (1) 
Depth of public discontent (1) 
Determining what is minimal (1) 

Con£>arisori of graduated students from MCT and non-MCT schools (1) 
There was little consensus on any important information that was 
omitted. The most mentioned comment was that there needed to be more ^ 
explanation about the variety of MCT programs and the laws that have 
produced ,MCT programs. These comments included statements that the 

i 

Clarification Process did not differentiate between MCT. programs with 
state mandated vs. locally adopted tests. 



* Throughout this report/ numbers in parentheses behind statements in 
lists refer to the number of responses mentioning the point. 
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Question le: Were there more Important issues that wjare not 
addressed by the MCT Clarification Process? 
Three issues were chosen to be addressed during-v£he MCT Clarification 
Process. There was concern about whether these we're the most important 
issues and if others that were more important should have been 
addressed. To obtain this information, both the hearing and site visit 
audiences were asked to identify other issued that could be considered 
more important. Fifty-eight percent of the audience responded that there 
were not any more important issues. The//audiertce at theJtxaaarirSg^rovided 




a range of responses that included ' issuesi^Tthe three stlpul^ted^ 



agreements, test content-applicability to life skills and effects of 



labeling in self-esteem. None, though, stood out as a compelling issue" 
that should have been addressed. From the states, there was also very 
little consensus and the six most mentioned issues were : extent, 
structure, quality and effects of remediation; how to set standards; life 
skills vs. basic skills; acceptance of tes^ results between' districts; 
follbw-up studies on graduates; and the economics of decisions. 

The Clarification Process addressed the most important MCT issues and 
did not emit important information based upon responses by hearing and 
state visit respndents. v 
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Question If; Overall, what were the strengths and weaknesses of the 
Clarification Process? 

One of the last questions about the shows concerned the strengths and 

< 

weaknesses of the Clarification Process. The major comments from the 
( •■} audiences concerning the; strengths were: 

Number of Comments 

Highlights the major issues and gives a sense 

of the pro and con biases 1 , (32) 

Presents a variety of persons and opinions from 
various regions and experiences. (The most 

mentioned category at the hearing.) (27) 
More interesting and will make a better impact 

than written material (22) 
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Involved expert witnesses and people who have 
a sense of personal, emotional commitment to 

a viewpoint (16) 

Good use of limited time (?) 

Lively, exciting, dynamic presentation (7) 

The give-and-take promoted discussion c (4) 

Dramatic quality (4) 
Allows for probing questions and 

cross-examination ( 4 ) 



The audiences were also asked to identify the weaknesses of the 

Clarification Process. The major comments concerning the weaknesses were; 

/ ■ ■' ' 
/ 

Number of Comments 

Personalities can be more persuasive than 

facts (14) / 



Opinions without data and documentation (13) 

- ." * < •. 
Lack of structure and wavering from the issues 

(also mentioned at hearings) (12) 
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Number of Comm ents 
Need fewer people, more experts, more depth (11) 
Variation in quality of questioning and 

cross-examination (8) 
Editing eliminates some clarification, 

truncates arguments (7) 
Dichotomizes the argument; avoids middle 

ground (5) 
Desire to win not necessarily to present 

valid arguments (5) 
Superficial, oversimplif ication' of a 

complex topic . (4) 
Lack of definition, of MCT; not uniform 

nationwide / ~ (4~)~ 



Allows for pbsMble inaccurate statements ( 4 ) 

Too long (4) 
In summary, the format and structure of the Clarification Process 
were generally viewed as appropriate in presenting MCT issues* 

j 

Appropriate is best interpreted, based upon comments, as presenting 
information Which highlights important MCT issues, makes good use of 
personal judgments and provides an interesting format for presentation. 
Factors such. as direct/cross-examination, specifically, or general 
adversarial format, in general, were not cited as particular strengths or 
advantages for the Clarification Process. The strength of the 
adversarial process might be that presenting pro and "con so that "both 
sides are heard" enabled the audiences to benefit from interesting 
personal testimony with some check on a major weakness, i.e., 
personalities can be more persuasive than facts. Finally, the number and 
nature of strengths and weaknesses which were cited indicate an overall 
positive reaction to the Clarification Process with some concerns or 
reservations. 
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2. Did the information presented represent a fair diversity of 

viewpoints on each issue that was seen as clarifying and illuminating? ^ 



Question 2a: Did the Clarification Process offer a variety of 
viewpoints on MCT? _. " 

o 

Both the hearing and state audiences were asked to agree or disagree 
to a statement that the Clarification Process offered a variety of 
viewpoints. For the state audiences/ the question was asked after each 
show. The results of this question are represented in Table 6. 

There is no doubt/ based on the data in Table 6, that the audiences • 
-^1:t-that they were being presented with a variety of viewpoints with the 



average around 3.2 ( Agree=*3 ) * At least 81% of the responding audiences 
agreed with this statement and this was a frequently cited strength as 
well. The MCT Clarification Process certainly achieved this goal. 
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TABLE 6 

Did the Clarification Process Offer a Variety of Viewpoints on MCT? ■', 
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Question 2b: Did the Clarification Process provide an opportunity 
for a fair discussion of the MCT issues? 

Once again a statement to this effect was made and the respondents 
were asked to agree or disagree. This question was asked of the 
site-visit audiences at the end of each show and in the final questions 
concerning all three shows. The results are summarized in Figure 5 and 
in. Appendix E by state. 
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Figure 5. Audience responses to The Clarification Process Provides an Opportunity 
for a Fair Discussion of MCT Issues. (Responses shown in percents.) 
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The audiences at the site visits,, were also asked the question after 
6 each of the shows. The results suggest that most persons viewing the 
tapes (81%) and the hearing (79%) saw the opportunity for fair 
discussion. The most negative responses came from the California group 
where '37% disagreed with the statement and the most positive from Texas 
where all respondents agreed. , 

Based on these responses, a major objective to present a fair 
discussion of the MCT issues was accomplished. It should be remembered 
that one of the major reasons for undertaking the Clarification Process 
was for the NIE to sponsor a study which would not appear o to support or 
decry MCT, but would provide information in a fair manner so others could 
make MCT decisions. 
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Question 2c: Were the arguments presented clearly by the pro and con 
teams? 

Respondents at the hearing and after each show at the site visits 
were asked to agree or disagree with a statement that the two .teams 
presented clear arguments. The results are summiarized in Figure 6 and 
in Appendix E by state* 
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Figure 6. Audience responses to Arguments were Presented Clearly by the Teams. 
(Average responses shown.) 
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The edited tapes produced a more favorable response to the con teams' 
arguments than did the hearing • There was around a half a standard \ 
deviation difference between the rating of the pro and the con argument^ 

i 

at the hearing with the audience stating * that the pro team presented ; - 
clearer arguments* This difference was not quite as great in the opinion 
of the state audiences, even though the pro team rated consistently 
higher than the con team on all three shows* 

The objective to have clear arguments was achieved- Even the lowest 
rating of 2.8 for the con team's arguments at the hearing is evidence 
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Question 2d: -Was there any information that you feel was not useful 
in clarifying the MCT issues? - 

The audiences at the site visits were asKed to list the information 

in that show which they felt was not useful. Following isr-a list of the 
most frequent responses to this question. . . 



Show 1 



Number of Responses 



/ Subjective ("I feei type") testimony 
Attack on multiple-choice tests 



15 
10 



/ 



/ Show 2 



/ 



Special education parent 
Standard setting and technical material 
Redundant testimony 



Show ' 3 



Handicapped issue 
Subjective testimony 




It is interesting that even though on previous questions the audience 



felt that the personal testimony enhanced the presentation of issues and 
offered a variety of viewpoints, it was the one element that paused the 
most concern when it came to useful information. - This same concern 
appeared in the interviews after the shows. 

Al^so, the - attack on multiple -choice tests, although considered valid | 
by some, did not seem to be an MCT issue to the audiences during the 
interviews. Attacking mulciple-choice tests did not" seem equivalent. to 
attacking minimum competency tests. / ■ 

The listing of the special education students' parents in the second 
and third show is' indicative that members of th<» audiences felt that 
special cases should not be used to attack MCT programs in^general. 
Others in the audience seemed to find the use justified for the purpose 
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of alerting viewers to potential problems that could be faced or needed 
to be avoided. 

In summary/ subjective testimony or personal judgments are viewed to 
add interest and to clarify how individuals "feel" about their MCT 
programs but doubt exists as to the value of the information in 
clarifying the issues. It appears these personal judgments are viewed as 
best for highlighting rather than clarifying issues. I 
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3. Did the information \presented add to the current understanding and 
knowledge of Minimum Competency Testing? 



Questiv 



Did the audiences gain any new knowledge from the 



Clarification Process? 
The audience at the hearing was a3ked to rate their knowledge of MCT 
previous to the hearing an<i then to rate the amount^ of knowledge gained 
from the hearing. At the site visits , the audiences were also asked to 
rate their knowledge and were then asked after each show how much new 
knowledge they had gained. The results are displayed in Figure 7 for . 
initial knowledge and in Figure 8 for knowledge gained. Table 3A in \ 
Appendix E contains summaries by state. 
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Figure 7. Audiences' initial knowledge of MCT. (Responses shown in percents. 
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Figure 8. Audience responses to Did You Gait) Any New Knowledge About MCT?. 
(Responses shown in percents.) 



Almost all of the audiences had at least some knowledge of MCT before 
attending^either the hearing or the state meetings. The audience that 
was the least knowledgeable about MCT was Montana where the MCT issue is 
just beginning to ke discussed. At least half of the site-visit 
audiences gained new knowledge from each of the shows but the number 
decreased from 63% saying at least "Some New Knowledge" for the first 
show to 45% for the third show. This is substantiated by interview, 
comments about the redundancy of information contained in the third show. 
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Question 3b: What was the most important new knowledge gained? 

After, each show during the site visits, the audiences were asked to 
provide the most important new knowledge that they had gained from that 
3how. The major comments are listed below: 



Show 1 



Specific information about particular MCT 18 



programs 



MCT may lead to less local control ) 5 

Effect pn curriculum - standardized vs. [ 3 

/ 



individualized 
Show 2 



Specific practices and results of state and 9 

local districts 
Impact on special categories of students 8 

(handicapped, Non-English speaking, mobile) 
Chance of undue failure on repeated testing 7 

small / 
Concerns of test validity / 3 

Relationship of MCT to life success / 3 



Show 3 



The variation among MCT programs / 3 

38 states already have MCT 3 j 

- Effect on "high risk" students 3 f 

From these responses and from the comments made during the po^t-show 
interviews, it was obvious that some of the audience was interested at 
first in information about how present MCT programs operate and how^they 
differ across the country. This in turn has implications for training 
sessions using the tapes. A presentation that involves using the tapes 

• I' ■ 

could definitely benefit from some preliminary orientation to existing 
MCT programs if the audience is similar to those present at the shows. 
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Question 3c s Was any information presented that was inconsistent 



wi th what the viewers know about MCT? 

After each of tho shows in the site visits, the individuals were 
asked to list any information that vas presented in the show that was 
inconsistent with. what they knew about MCT. The reason for this question 
yas to aid in judging the validity of the information that audiences were 
viewing from the. tapes. Listed below are all the comments mentioned 
after each show.V 

Show 1: Inconsistent Information 



• / 



Not aware of areas where MCT is the sole ( 6) 

criterion of pass/fail 
Impression by con team that only one test is (4) 

given when in reality testing is begun in 

9th or -10th grade and"repeated~yearly-*f br * 

remediation 

Con team chose some of the worst examples ( 2) 

Why so much testimony related to MCT as a (2) 
retention/promotion instrument when the 
pro and .con teama agreed that MCT should 
not be used for x "ition? / 

California wants MC ,o be used to determine / ( 2) 
diploma eligibility to function as an early/ 
warning system so appropriate remediation , 
can be given - j 

No discussion of Calif or^a^-^ct^rcdise of (2) 
having local districts set their own 
standards 

Education jLs political 

Students are informed of requirements in 
New York. 

Without testing, teachers do not know who the 

low students are . 
South Carolina does not have MCT 
Tests are available for inspection (Nader) 
Ralph Nad - is not an authority on competency 

testing 
MCT is r"ef uteable 

Ralf>h Tyler talked about a state mandated MCT 
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show 1 ( cor.it • ) 



Ralph Tyler has advocated state mandated MCT 

programs and then he comes out as CON? 
Our district tests writing by, writing, not 

multiple choice 
Process of test development is not secretive 
Minimum tests become maximums (Benton) 
No statement regarding baseline quality of 

education in South Carolina was provided 

to substantiate claims 
Dr. Cronin was somewhat inconsistent between 

verbalization and actual practice . 
Interesting that remediation was not always 

linked to the MCT program 
Distinct failure t6 distinguish between 

competency based education and MCT 
MCT does not automatically cause an -instructional 

program to improve (Pro team argument) 
Abuses of MCT generalized to everywhere 
Standards do make provisions for measurement 

error 

Show 2 : Inc onsi ste nt Inf ormat i on 

Solely necessary for graduation ( 
Special education kids would take same MCT as ( 

prescribed for general students 
Inference that tests are translated into other ( 

than English 
The chance of a student being continually, 

failed simply due to error of measurement of 
r • a given test. Repeated offerings of MCTs 

reduce measurement error to insignificant 

levels; too much made of a moot point 
Children must past test in spite of never 

having a math course with a certified math 

teacher 
Bias statement by Ebel 

What caji be clearly tested is usually trivial^ 
Testimony by Berry—It was a biased / 

study led by armed opponents to MCT V 
Students who fail portions of MCT can't take ^ 

other relevant courses like literature and 

calculus (Schmidt) 
Florida section was grossly misrepresented 
NEA does not represent Flordia teachers 

feelings about, this issue* 
Multiple choice tests are the standard for MCT 
Amount of testing done today (Farr) 
False assumption that all MCT programs result 

in pass/fail decisions about promotion/ 

retention and graduation 
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Show 2 (cont. ) 

John Myrick saying that this test, MCT would mean 
too much testing. To me this would displace 
some of the other testing, not add to it^ 

Do not think most tests contain items with 
words not used in any other tefits 

Individual cases are extremely weak basis for 
establishing generalizations 

Distinct failure to distinguish between CBE 
and MCT 

High degree of passing in second test 
Special Ed. mother-- don't believe two weeks 

ample time to prepare for retake of test 
What some groups claim they measure 
Some of the states ' using MCT have reportedly 

not gained what they would profess 

Show 3 ; I Inconsistent Information 

. Teaching the test 
Differential scoring 

Many of the arguments against MCT are of a 

regional basis and have been addressed in 
r , other areas 

Failure on MCT is only one criterion — true; 
but it does have veto power 

Madaus — in closing argument that 11,000 more 
Florida kids would have gotten a diploma had 
the cut score \been lower — no Florida kid was 
denied a 1 diploma 

MCT is not predictive; it measures special 
skills A s 

The CTBS i.s a competency test (Perna) 

The test is not constructed to fail a number 
of students (Bracey) \ 

The test error or measurement. error concerns 
are not as much a factor^ as depicted for any 
of the MCT programs where test validation 
practices were adequate arid where repeated 
testing is offered \ . 

MCT will not solve racial problems 

Change in school participation* with parent 
participation teach expection not directly 
tied to increased competency \ 

There were testimonies still about special 
education kids that failed long after 
everyone conceded they (the special \ 
education kids) should not take the same 
' test ' \ 

Sincle MCT concept. Reality — numerous 
variations 
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Question 3d: What were the moat persuasive points made by the pro 
team? The con team? 

In order to see what points the audiences were picking up in the 
hearing and the shows, a question was asked which had them identify the 
most persuasive points made by each of the teams'. 

From the hearings * Responses to pro and con team points were clearly 
targeted to issues of effects on students. Statements about the pro team 
also frequently cited effects on curriculum and public perceptions of 
education. Statements about the con team did not as often contain 
evidence for effects on curridulum or public perception, but appeared to 
focus more on specific effects or factors about MCT, e.g., cut-score 
unreliability and other cautions about using tests. Other points even 
though not frequent from the con team, appear to be that resource 
allocations on curriculum/improvement strategies would be better than 
resources goinc; to MCT. Combined pro and con team points might be 
characterized as providing information about (a) potential for MCT given 
.some actual cases of positive MCT programs, and (b) pitfalls to avoid in 
undertaking an MCT program. , Little evidence was gathered to suggest that 
strong information abo\>.t how to implement a program was presented. 

From the state visits . Below are listed the major points that were 
mentioned by the respondents : 

Show 1 
Pro 

MC' * itifies individual needs for 31 
repudiation 

MCT programs inrfrcas'5 accountability, 26 

credibility and public confidence , 
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Show 1 



Pro (cont.) 

MCT focuses objectives and improves the 
curriculum 

MCT assures that necessary skills are being 
mastered 

Con 

No single test should be a sole criterion 

Tests and test items are imperfect 

MCT reduces local control of curriculum 

Tests do not match what is taught 

Tests are redundant sources of information 

Minimums can become maximums 

Show 2_ 

Districts can have positive results 
MCT increases accountability, credibility , 

and public confidence 
Passing levels can be made with informed 

judgment 1 
There is a better chance of students receiving 

instruction and remediation 
Minority children also heed the skills 
Making choices (as in multiple choice tests) 

is a way of life 
There is a low chance of repeated undue 

failure / 
Teachers should be involved with MCT 

development 
Expect high, get high 

Con 

MCTs are culturally biased 

Danger of misclassif ication and labelling of 

students 
Curriculum can be narrowed 
Tested skills do not imply success in life 
Arbitrariness of cut-off scores / 
- Evaluation should not be based on a single 

criterion 

Tests often do not have curricular validity 
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Show 3 
Pro 

Some school districts have been successful 

upgrading / 
MCT increases accountability , credibility and 

public confidence 
Minorities support MCT 
There is an increase in positive attitude 7 

0 

Con 

Schools can improve without MCT 21 
Tests determine the curriculum 9 
There will be teaching to the exams 6 
Cut-off scores are arbitrary 6 
Decisions should not be based solely on the 5 
test 

There needs to be special treatment for the 5 
handicapped 

Specific knowledge gained from the pro team's case appeared to be 
easily related to the three hearing issues. Individuals felt that the 
pro team points included that MCT could (a) assist to identify students 
in need of remedial instruction, (b) facilitate in assessing individual 
students' strengths and weaknesses, and (c) reduce discrimination in 
schools by identifying students' educational deficiencies and thus 
revealing any potential discrimination which led to the deficiencies. 
Effects on curriculum were delated to arguments that (a) MCTs were useful 
for identifying instructional objectives, (b) assisted to focus 
curriculum across schools, districts or a state, and (c) standardized 
instruction resulted in more efficient remediation of basic skills and 

thus greater flexibility in other areas. Finally, the above points 

/ 

appeared to be somewhat persuasive in convincing viewers that MCT would 
have a positive influence on the pMblic perception of education. An 
example is several viewer comments related to the fact MCTs might assist 
to reduce discrimination and improve educational programs. 
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Information gained from the con team's case can also be classified 
around effects on students, r.urriculum and public perception of / 
education. However , the points were addressed through a different 
perspective and were not seen as clearly related to the hearing issues, as 
the pro team's case. Arguments about effects on students focused on 
negative effects of labeling students who fail MCT, effects of unreliable 
tests and cut scores and effects due to lost educational opportunities if 
students were retained in grades or denied high school diplomas. Effects 
on the curriculum included reduced curriculuar emphases due to overly 
focused attention on rainimums, teaching to the test vs. teaching toward 
skills or objectives and reduced number of courses available as a result 
of the increased number remedial classes. Effects on public perception 
appeared to center on the above issues and that MCTs would not by 
themselves have a positive impact on education or the public's perception 
of education. 

Although the cases can be related to the three hearing issues , 
comments from both the edited videotapes and the hearing indicated that 
the audiences did not perceive the pro and con teams to address the sair - 
issues, and that neither team was directly addressing the issues. Thus, 
although subsequent analyses of the oases and testimony reveals that it 
is; possible to organize the teams 1 cases around the issues debated in the 

i 

Clarification Process, this was not apparent to the audiences. This' is 
probably a characteristic of the personal testimony approach to 
presenting information. This approach is characterized by individuals 
giving their reactions which generally cut across issues. In the pro 
team's case,, the general impression often was that MCTs are good, have 
helped focus programs and generate public support. The con team's ease 
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was more often perceived as focusing on technical issues such as the 
effects of using a single test for high school graduation or grade 
promotion/retention. The pro team f s contention that MCT was only one of 
several criteria did not result in a clarification of this specific issue 
since the audience expressed confusion about^the issue, i.e., individuals 
felt no case was made that the MCT was used as a "sole criterion."' In 
this case the audiences felt that the pro team had "defeated or won" this 
point by showing that the MCTs were never used as a sole criterion. The 
purpose of this example is to illustrate that the viewers gained 
knowledge which was not necessarily related to the specific phrasing of 
the issues debated. 
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Question 3et Who were the moat effective witneaaea for t.ha pro 
team? For the con team? 

In an effort to analyze the strength of the two teama argumenta, the 
audiencea were aaked to identify which witneases they thought were the 
moat effective for each team. Tile queation was asked after each ahow 



during the aite viaita. To help the groupa keep track of the witnesaes' 
names, they were given a list of Witnesses names that appj^Qd^on^each 
show. Below are listed the most effective witnesses for each of the 
shows* 
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Most Effective .Witnesses \or Pro and Con Teams 



Show 1 



Pro 



Con 



Michael Scriven (57) 

Director of The Evaluation Institute , University of San 

Francisco * 
Jamps Popham (14) 

Pro team leader 
South Carolina educators (13) 
Paul Sandifer (11) 

Direptor of Research, South Carolina Department of Education 
Laurie Collier (8) - 1 

Supervisor for Business Education, Newport News Schools, / 

Virginia A 
Joseph Murra^s.(6) 

State Legislator, South Carolina , 
Gary Leonard (6) 

Principal, Mt. Pleasant Academy Elementary School, South 

darolina 

Virginia' Witness (Unspecified) (6) v - 

Deborah Meiers (51) 

Principal, Central Park East Elementary School, New York City 
Ralph Nader (44) -^JR* 

Director of Tne Center for Responsive Law, Washington, D. C. 

Ralph Tyler (38) 

Consultants Science Research Associate, Chicago, Illinois 
, Arthur Wi%cM-21) f 

Senior Social Scientist, Rand Corporation^ 
Gilbert Austin (11) 

Co-Director , Center' for Educational Research, University of 

Maryland y 
Joseph Cronin (8) 

Former Illinois Chief State School Officer 



1 

Show 2 



Pro 

Robert. Schilling '(45) , 
( * Superintendent, Hacienda La- Puerta District , Calif ornia 

V Robert Ebel (42) 
^v£rpfessor of Education, Michigan State University 
,Ralph TuflingtcyQ^(26) 

Florida Commissioner bf Education 

W Trujillo (10) , • ^ 

uperintendent, Mt. Tamplpair Union High School District, 

»Calif ornia ,/ 
Mor ri s Andrews ( 7 ) 

Executive Director, Wisconsin Education Council 
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Con 



Robert Calfee (34) 

Professor, of Education, Stanford University 
Robert Linn (24) 

Department of Educational Psychology, University of Illinois 
Roger Farr (17) 

Professor of Research in Education, Indiana Univeristy 
Patricia Shea. (16) 

Mother of Handicapped Child, Peoria, \ Illinois 
Lorenza Schmidt (15) 

State School Member, California 
Sliirley Chisholm (10) 

U. S. Congresswoman , ' ' 

Claire Sullivan (8) 

Florida Association of Supervisor and Curriculum Det^elopmerit , 

President 
Mary Berry j (6) 

Commissioneryand Vice Chariman, U.S. Commission on Civil Rffc 
Natjian QuiAones (6) 

Executive Director, Division of High Schools, New York .City 
Mel Hall (5) 

Program Director, Sangoma Univiersity, Illinois 



Pro 



Con 




Show 3 



William Raspberry (33) 

Syndicated Columnist, The Washington Post 
Detroit Educators (19) 
Clara Rutherford (i9) 



Detroit Schools School Board Member 
Craig McFadden (8) / . 

Director of Psychological Services and Testing, Goldsboro City 

Schools, North Carolina 
Michael Priddy (8) 

Director of - Research, Planning and Evaluation, Guilf ord County 

Schools, North Carolina / 
Stuart Rankin (8) 

Assistant Superintendent, Research, Evaluation and Planning, 

Detroit /. , 

Linda Spight (7) 

Test Coordinator, Henry Ford High School, Michigan 
James Popham (6) \ 

Arthur Jefferson (5) \ \ 

Superintendent, Detroit, Michigan,\ Schools S 

• ~ v ••■ \ 

Gerald Bracey (20) * ' 

Director of Research, Evaluation, and Testing, Virginia 
Department of Education 

Frederico Penna (18) 

Attorney and Colorado State Legislator 
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Show 3 

Con (cont. ) 

Henry Stevens (11) 

Teacher, Camden Public Schools, New Jersey 
William Shine (10) 

Superintendent, Washington Township Public Schools, New Jersey 
Kathleen Gilbert (10) ^ ^ 

Teacher, Hope Valley Elementary School, North Carolina 
Esther Lee (10). 

Title I teacher, Camden Public Schools, New Jersey 
George Madaus t (5) 

Con team leader 
Lawrence McNally!.(5) 

Director of Public Services, North Port-East Port School 

Dintrict, New York 



The witnesses who were rated as most effective presented either 

persuasive points (Question 3c) or knowledge gained (Question 3a). It is 

also interesting to note that the pro team leader, Jim Popham, was rated 

as an effective witness by 14 viewers. To a lesser extent, witnesses not 

' \ \ 
rated as effective were identified with inconsistent information 

• " m ' 

(Question 3d). These data would be useful if further editing were to be 

done to produce a one-hour- tape, or if one were to use only particular 

segments of the tapes rather than the complete three-hour set. 

In summary/ the Clarification Process clarified or^ provided 

clarifying information related to MOT. This statement is supported by 

comments related to inf ormat ion gained and persuasive points made.. . The 

statement is also tempered by the fact that viewers cited numerous cases 

where they believed misinformation .was relayed. Although there was no 

^'consensus as to the misinformation, findings here support our personal^ 

judgments. Specifically, viewers indicated that individuals' testimony 

did leave some misimpressions. Finally, most ^individuals responded that 

they gained new information. This information is best characterized as a 

1 
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3ynthesi« and overview of what exist, within MCT programs* Little 
evidence was observed that the Clarification Process added to the body of 
knowledge on MCT. The objective of the Clarification Process was to 
provide individuals' with existing information on key MCT issues. This 
objective was met and appears to be a viable objective for the 
Clarification Process approach. Other approaches will probably be as 
effective when the study or research is primarily intended to add to a 
knowledge baSe or to discover new facts. 
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4 # Did the audiences perceive the information- to be useful in terms, of 
pending policy or program decisions? 

\ 

Question 4a; What was the audience's opinion of MCT before the 

73 

hearings or tapes and what change occurred. If any? 
Audiences both at the hearings and at the site visits were asked to 
rate their opinion of MCT before and after seeing the MCT Clarification : 
Process. The results from these questions are summarized in Figure 9 for 
the hearing audience and in Figure 10 for the state visit audiences. 
Detailed responses for states are in Table 4A. in Appendix E. 



» 100% 
90% 
80%- 
70%. 
60%- 
50% 
40% 
30% 
20% -| 

10% 

' 0 



Before Hearing 
IM-62 



After Hearing 
N = 61 



21 
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32 



11 



24 



48 



13 



Figure 9. Hearing audiences' resvnses to How Would You Rate Your Opinion of MCT? 
(Responses shown in percents.) * 



\ 




9 

ERIC 



83 



100 



100%! 

90% 
.80% H 
70% 
60% H 
50% 
40% 
30% H 
20% 
10% 
0 



;. \ 

Before Shows 
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Figure 10. Site audiences' responses to How Wouid You Rate Your Opinion of MCT? 
(Responses shown in percents.) 



Both at the hearings and at the state visits, in general, the opinion 
of MCT remained constant or slightly improved. It should be remembered, 
however, that not all the persons at the site visits were able to stay 
for the whole day. Montana respondents were much more favorable after 
viewing the shows. This is especially- interesting since Montana is just 
beginning to consider MCT. In states where programs have been instituted 
to some degree - California, Florida, and Illinois - the opinions did not 
change appreciably. In Wisconsin, where people have been discussing the 
issue of MCT for awhile and where a decision is about to be made, it is 
not surprising to see little change from the initial opinion. In Texas 
it is difficult to determine from the small sample size but the tape v s may 
have made a bipolar situation even, more so. 
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It was not anticipated that viewing the tapes or attending the 
hearing would substantially change individuals' opinions' for or, against 
MCT. Instead, these data are primarily intended to provide descriptive 
information about opinions toward MCT across and within states. An 

interesting observation at -the hearing which was not- observed across 

j * 

states was the decrease in the tendency to rate neutral one's opinion of 
MCTo Another interesting finding is thai: no one in Florida was opposed 
to MCT. It is clear that a substantial sector within Florida does oppose 
MCT— at least in its current form in Florida. Thus, although most 
respondents in the study favor MCT, the general population is probably 
less in .favor. 




85 ' 

102 



Question 4b: What MCT issues were the audience most concerned about 



and would the information from these shows be useful in relation to 
these issues? 

In the site visits, after all three shows had been viewed the . 
audience w*s asked to identify what MCT issues were of most concern to 
them and if the information in the shows might be useful to them. Listed 
in Table 7 are the major issues that the audiences listed as being most 
concerned about* 

Table ,7 

What MCT Isaues Are You Most Concerned About? 
(Five or More Responses) 



Impact on curriculum 




17 


Diploma attachment 




11 


Remedial program 




10 


Cut-off scores/standards 




8 


Loss of control for development 




8 


Criterion validity 




7 


Impact on minority students 




6 


Use or misuse of MCT results 




5 


Emphasis on one test 




5 



After identifying the issues that were important tj> the audiences, 
they were then asked if and how the information would be useful to them. 
Listed below are the comments to this question. 
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Summary of if and how information will be useful to audiences 



Being , aware of what practices axe going on (6) 
Identifying issues r f or discussion (6) 
Informing public about processes and practices , (6) 
Staff development workshops (4) 
To provide questions and discussion among educators (3) 
Provides persons and places who can serve as (2) 
resources 

In countering the "con" view (2) 
In presentations against "pro" 

Educate public about shallowness of the opposition 
In re-evaluating our tests 
Evidence to cite / 

Identified outcomes as related 'to both curriculum and instruct. ion 
Vivid examples ror recall 

In dealing with school administration and teachers 

Districts using the tests discussed methods for involvement with 

positive, results 
No--only raised issues I already knew about but didn't provide a 

basis for decision-making 
Texas has not mandated that performance on its assessment be tied 

to graduation promotion. This information will be helpful if 

that issue comes up on a statewide basis or as local districts 

may elect to pursue it. 
The opinion that minority students will do and achieve what is 

expected and taught to them. 
Minimally because level of presentation was not aimed at the 

technician. 

Become more active to see that Illinois retains local control 
policy. Arguments for local controx are strengthened due to 
fear of centralization / 

"'From the comments, the information will be primarily useful for 

1' 

making persons aware of what is presently happening and the issues that 
need to be addressed. It will also be useful for stimulating discussion 
in meetings and workshops. Few responses indicated that the information 
was directly useful in making a decision or choosing a direction. 
However, the statement "I plan on becoming mpre active to see that 
Illinois retains a local control policy" is an example of a decision or 
position, facilitated by the Clarification Process. Again, it is positive 
that individuals who viewed the tapes felt they were useful for awareness 
or general discussion purposes. Seldom are decisions or judgments based 
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upon "revelations" obtained from a single experience or source of 
information, but the information from the Clarification Process can 
assist with discussions/ deliberations or decisions about MCT. 
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Question 4c t W as the audience at the hearing interested in using the 
throe one-hour edited tapes? 

One of the questions on the questionnaire distributed at the hearing 
asked how interested the audience would be in using the three tapes that 
were going to be edited from the complete hearings tapes* Fifty-five 
responses were obtained with 13 (22%) very interested, 29 (49%) 
interested and 14 (24%) not interested in using the edited tapes. This 
finding is fairly consistent with the responses obtained in selecting 
states for the site visits in which 12 states were contacted to obtain 
six which were willing to participate in viewing the videotapes. 
Similarly, it appears that approximately half the PBS stations will air 
the shows. Thus, 3ome interest in the information from the Clarification 



Process does exist even though some 



form of MCT is established in 



approximately 38 states. The qualifier "some" is used intentionally. It 
is significant that five states werja not interested for various reasons, 
and the times when the PBS broadcasts will occur are clearly not prime 
time. The PBS broadcasts might best be described as opportunities for 
others to tape for later use rather than the prime target being the PBS 
viewers (see PBS section for fuller discussion). 
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Question <li How uooful will the information from tho hearings and 
ahowa be for various function a? 

Four possible functions were identified Jfor tho information from ^tho 
hearing and shows. Viewers wore asked to rate tho usefulness of the 
information for performing four functional 

1. Formulating a policy about adopting an MCT program 

2. Assisting with the implementation an MCT program 

3. Informing the general public about MCT 

4. Revising or dropping an MCT program 

(Tho fourth function was rated only on the questionnaire given at th 
site viaits. ) Figurea 11 and 12 contain responses for the hearing and 
state audiences, respectively. Detailed summaries are contained in 
Table 4d in Appendix E. j 
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Not Useful Highly 
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Figure 11. Hearing audiences' responses to Rate the Use of the information from the 
Hearing for Different Purposes. (Responses shown in percents.). 



90 



107 



90% 

80%- 

70%- 

00%. 

50%- 

40% 

30% H 

20% 

10% 



Formulotlng 
Policy 
N = 122 



Assisting 
Implomontntlon 
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Figure 12. Site audiences' responses to Rate the Use of the information from these. ■ 
\ Shows for Different Purposes* (Responses shown in percents.) 

This very important in that it reflects the overall utility of the 
product for the states. Some very interesting patterns can be seen from' 
Figures 11 and 12. 

r 

First the audiences in the states and at the hearing saw the 
information more useful in the areas of formulating policy and informing 
public, and less useful in the areas of assisting implementation and 
revising or dropping a program. For areas of* general informational need 
the audiences saw the information as useful. When specific information 

is needed/ such as in the implementation or revising/dropping functions/ 

- \ 

the information was not seen as useful. 

It is even more interesting to study the distribution of opinions 
between the different states. Views vary depending on the level of • 
implementatic-n currently in each state* The reactions of the audiences < . 
in Califoria and Florida were typically lower than those in the other 
states. 1 For the sake of simplification/ the states have been separated 
as to level of implementation and the weighted means are presented in 
Table 8' ' 
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TABLE 8 



Use of the Clarification Process 
Weighted Mean Responses of. States 
Categorized by Implementation Level 

3 ■ Highly Useful 
2 - Useful 
1 ■ Not Useful 



Formulating Assisting 
Policy , . Implementation 



Legislative mandate 



M« 1.9 
N- 42 



1.5 
41 



California 



Florida 



MCT in- use but not mandated 
Illinois 
Texas 

r 



M- 

N» 



1.7 
35 



25 



1.9 



Pending decision 



Wisconsin 



M- 
N- 



22 



2.3 




Under consideration 



N- 



2.2 
25 



1.9 



Montana 



24 



• : From Table 8 it"can-^e seen that Formulating Policy and Informing 
Public still rate the lowest ^mong the four function categories, but the 
states now having legislative maAdates usually gave lower ratings in all 
four categories. Consistently higher ratings were from Montana and 
Wisconsin where MCT is under consideration. The Informing Public 
category was rated highest across all four levels. of MCT implementation. 
Apparently, information from the hearing is more useful at the beginning 
stages ofjbhe decision-making process >and for general information than 
for revising existing programs or making implementation decisions on 

i ■ 

policies. • 

In summary, the information from the 'clarification Process was rated 
as useful for general information and discussion. Interest exists in 
using the information from the videotapes based upon responses from state 
visits, the hearing and the PBS survey. Thus, the Clarification Process 
is effective using the qriterion outlined in our introduction, i.e., the 
information should be useful as one source of influence in decisions or 
deliberations. . 

■ / 

One caveat is needed in concluding that the Clarification Prpcess 
? ' 
accomplished the objectives outlined in Chapter I.' Specifically, a hi^h 

v - ■ 
level of "self-generated" interest in using the Clarification Process - 

information has not been evidenced to date in information obtained within 

this evaluation^ Six states declined to participate in a one-day meeting 

and within states, individuals did not /express overwhelming interest. In 

summary, opce the information is viewed, positive responses are made. It 

is important that dissemination type efforts such as those initiated by 

/the NIE^b^ continued and that support, fiscal and personnel, be provided 

for these efforts. - 
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5. What is the viability of this approach for other NIE efforts? 

Question. 5a; How does the video presentation format compare to 

written evaluation reports? . 
The Clarification Process was chosen as an alternative to the more 
traditional evaluation report procedur^ One viability concern was 
whether audiences would find the Clarification Process at least as useful 
as a written report. Table 9 contains a summary of the responses to a 
question presented on the state visits questionnaire. \ 
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TABLE 9 



Hov the Clarification Process Comperes to a Written Report 



Not as good About the same Better than 





(11 




(3) 






• 

■ 




LB 1 




US 


• 


• 


I 


SITES 












. j 


Los Angeles, CA 






12 (100) 


12 


3.0 


0 

1 


San Francisco, CA 


.2(18) 


1 (9) 


8 (73) 

1 


11 


2,5 ; 


' 8 


Florida 


3(33) 


2 (22) / 


4 (44). 


9 


2,1 


,? 


Illinois 


1 (8)/ 


' 2 (15) 


10 (77) 


13 


2,7 


,6 

j 


Montana 


1 (4) 


2 (8) 


21 (88) 


A 1 

24 


Ni ft 

2,8 1 


,5 

I 


Texas 






10 (100) 


X 10 


3,0 . 


0 ! 

>*\ 

J ' 


Wisconsin 


2 (10) 


2(10) 


16 (80) 


.. 20 


2,7 


,7 


Special Interest 
Groups 


1 (17) 


1 (W) 


4 (67) 


\ * 

- '6 ' . 


2,5 


" 1 
i 


TOTAL 


10 (10) 


10 (10) 


85 (81) 


105 


2,7 


i ' 

i,6 

1 



Number 



Standard 



X. 



/ 
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The results were c-verwhelmingly in favor of the video tapes as 
compared to a written report— 81% favored the video presentation. At two 
of the site visits/ Los Angeles and Texas , all tl-^ participants agreed- 
that the video presentation was better. The most negative reaction came 
from the Florida visit. 

Audiences seemed to feel that watching the video presentation was 

more interesting than reading a report. ' Their main concern was their 

inability to analyze and chsck the validity of the data for themselves. 

The negative response from the Florida audience could be becaus^ this 

question came after the third show. During the final arguments of the 

con teak in the third show, a strong point is made that if the cut-off 

\ , * 
score for\the Florida test was lowered by three points, 11,000 more 

^udents woxild have received their diplomas. The Florida concern stemmed 

fr<om the fact that no Florida V student to date has been denied a diploma 

based on the test. This led then to comments concerning the 

Clarification Process' proineness for "gross misrepresentation of facts." 

• For the parts of the audience that responded positively to the video 

presentation method, the nuftn emphasis was the appropriateness of the 

s for a variety of audiences that specifically included parents and 

school board members. It was felt they might be willing to spend a few 

hours viewing and discussing the tapes but would probably be unwilling to 

invest even the same time in reading a report. 

It is key to note that the viability and attractiveness of the 

Clarification Process rests largely on the fact that a video format was ^ 

rated superior to a written format in generating audience interest and in 

being useful for stimulating discussion on issues. This statement does 
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not negate the need for written materials like the User's Guide to . 
support the videotapes. The written materials address some. of the' 
criticisms above related to checking data*. Finally, it is less clear 
that the adversarial aspect of the Clarification Process was felt to be 
strong contributor to the interest in the materials. 
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Question 5b; What other educational issues do, you think could be 
covered effectively using the Clarification grocesa? 
One indication of the audience's reaction to the Clarification 
Process is to find what other educational issues they feel might be 
addressed in a clarification hearing* Below is a listing of the topics 



r 



mentioned by the audiences: 

.Bilingual education 
School finance 
Tax tuition credits 
Curriculum " issues 

Special education and rj&ainatraaming 

Teacher evaluation 

School discipline procedures 

Tuitiorf vouchers 

Teacher unions 

Busing and desegregation 

Federal intervention and local control 

Back to basics 

Gifted and Talented programs 
Vocational education 
Private schools 
Sexism in education 
Parental "involvement 
Compensatory education ^ 
Career education 
, Beginning reading 
Art and music education 
Education in the performing arts 
Sex education 

A School's role in. character building 
Methods, of instruction 
, Student grouping 
\ Computer-assisted instruction 
Competency-based education 
Teacher competency testing 
Teacher tenure 
Negotiations 
Block grants 
Differential staffing 
School organization 
\School closings 
District testing programs 
arogjr^ evaluation 

Predictive testing — SAT, GRE , LSAT 
Department of Education 
Community college issues 



(18) 

(13) 

(10) 

(6); 

(6) 

(6) 

(4) 

(4) 

(4) 

(3) 

(3) 

(2) 

(2) 

(2) 

(2) 

(2) 

(2) 
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Evaluation of the User's Guide 

One of the products of the MCT Clarification Process was a User's 
Guide designed to be used as a supplement to the edited tapes and the 

hearing. \> , "7 

The draft User's Guide contained the following sections; 
Preface 

I. Introduction (3 pages) 
Background and purpose of the guide 

II. Clarification Hearing: Description and Background (4 pages) 
Overview of hearing, its planning, development, goals and 
objectives 

III. The Cases: Pro and Con (13 pages) 

Outline of the cases presefifced by thG teams % 

IV. Discussion Guide (15 pages) 

Poses questions referring to specific testimony in the 

*■ -'.„■'■■' 

hearing that a decision should address 

V. Suggested Uses for Tapes and Transcripts (15 pages) 
Formats for using the. tapes and manuscripts 

VI. Summary' oO/itnesses Testimony (134 pa'ges) 

Summary of each person's testimony (categorized by state or 
area <of expertise') plus graphics^^esented as evidence at 
the hearing 
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, VII. Resource Guide and Index (29 pages) 

a. List of witnesses and location of testimony within the 
edited tape, the complete tapes, the hearing transcript, or 
the User's Guide 

b. References to documentory evidence introduced at the 
hearing 

c. Bibliography of references used by teams 

Respondents were asked whether the User's Guide was useful as a 
supplement to the videotapes. Table 10 reveals that 86% felt it was. 

A question from the site-visit^ questionnaire concerned the usefulness 
of the User's Guide for the same functions as were listed as possible, 
uses for the edited tapes and the hearings. The questions were only 
asked at five of the sites because the draft was not available for the 
first-three site visits with the special interest groups and the two- 
California groins. It should be noted that less than half the audiences 
responded to these itens. This response rate was likely a function of 
the minimal time available to review the User's Guide in a one-day 
session in which three hours of tapes were viewed and discussions were 
held about MCT and the Clarification Process. The results of this I 
question are shown in Figure 13. 
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Table 10 

Usefulness of User's Guide as "Supplement 



No Yes 
N % N % , N 

T 

Sites 

- ; 



Florida 


0 


(0) 


7 


(100) 


7 


Illinois 


1 


(11) 


8 


(89) 


9 














Montana 


2 


(14) 


12, 


(86) 


14 


Texas 


1 


(8) 


11 


(92) 


12 


Wisconsin 


2 


(24) 


13 


(77) 


i 17 


TOTAL 


8 


(14) 


51 


(86) 


59 



100%-, 
90%- 
80% -| 
70%- 
60%- 
50%- 
40%- 
30% H 
20%- 
10%-) 



Formulating 
Policy 

N-59 



12 



64 



24 



Assisting 
Implementation 

N-57 



74 



18 



Informing 
Public 

N-60 



27 



48 



25 



Little Useful Highly 
Use Useful 



Little Useful Highly 
Use Useful 



Little Useful Highly 
Use Useful 



Figure 13. Site audiences' responses to How Would You Rate the User's Guide for 
Different Purposes? (Responses shown in percents.) 
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The results of this question for the User's Guide differ from the 
results obtained when the audience was asked the same question about the 
edited tapes. The mean answer for all three uses is around 2.1 where "2" 
is "useful." Therefore the User's Guide was rated as equally useful in 
all three categories, although somewhat greater variability was noted for 
^responses to informing the public. Additionally, ratings of 2.1 for the 
User's Guide were higher than videotape ratings for assisting with 
implementation (1.7), revising or dropping (1.9) and were lower than 
informing the general public (2.4). 

This supports the conclusion stated earlier that the written 
materials si^>port areas cited as weaknesses about the videotapes. The 
audiences responded that the User's Guide would be a useful resource 
after viewing the videotapes and would be more useful than the videotapes 
in actually making .program or policy decisions. Some comments were that 
the User's Guide was lengthy and individuals wanted to check the 
representativeness of the summaries in the User's Guide with the 
presentations in the videotapes. 

Finally, a User's Guide or written materials should be included as 
products in any future efforts. Based on the state visits, it appears 
that one or at most two hours of videotapes will be sufficient and would 
allow more time for attending to and using information in the User's 
Guide for discussion or deliberations. 
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Information from Public Broadcasting Services (PBS) Stations 

One of the anticipated outcomes was that PBS stations across the 
country would broadcast the documentary and the three hearing tapes. 
They were originally scheduled to air September 17 and' the three k 
successive Thursdays thereafter. It was left up to the individual PBS 
stations whether they would broadcast the shows then, at a later time/ or 
at all. Questionnaires regarding the broadcasting of the shows and their 
comments concerning th± quality were sent by the Southern .Educational 
Ccmmunication Association to the PBS stations across the country. 

Twenty-eight questionnaires were returned. Of the 28, 19 (70%) 
planned to show at 'least the documentary . To get an idea of the 
representativeness of this sample, a random sample of 28 PBS licensees 
from the Directory of Information Sources for Public Television CPB were 
contacted to find if they had shown or were going to show the programs. 
Of this sample 14 (50%) responded positively. This implies that the 
sample of questionnaires seems to overrepresent the percentage pf 
stations that will air the programs. 

The stations were asked to identify the dates and times that they 
would be showing the tapes. This information is in Table 11. 
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TABLE 11 

ResponseB on Dates and Tines Shows Were Aired 



Respondent Documentary 



Program 1 



Program 2 



o 



2 
3 
4 
5 



10/13 (11:00 p.) 

10/17 (noon) 

10/25 (5:00 p.) 

TBA 

9/27 (7:00 p.) 
TBA 



q \ ■ Sunday . 
ERIC ■ Saturday 



.Program 3 



10/24 (noon) 7 10/31 (noon) 7 

V 

10/25 (6:00 p.) 1 11/1 (6:00 p.) 1 

TBA TBA 

10/4 (9:00 a.) 1 10/11 (9:00 a.) 1 



11/17 (noon) 7 

11/8 (6:00 p.) 1 
TBA 

10/18 (9:00 a.) 1 



Working vlth schools- 



m 

6 


9/17 


(9:00 p.) 5 


9/24 


(9:00 p.) 


5 


10/1 


(9:00 p.) 


5 


10/8 


(9:00 p.) 5 


7 


10/8 


(11:00 a.) 5 


10/p 


(noon) 


5 


10/9 


(11:00 a.) 


6 


10/9 


(noon) .6 


8 


, 12/20 
9/20 


(6:00 p.) 1 
(3:00 p.) 1 


















9 


9/27 


(3:00 p.) 


1 


10/4 


(3:00 p.) 


i 


10/11 


(3:00 p.) 1 


10 


. 10/3 


(5:00 p.) 7 


10/10 


(5:00, p.) 


7 


10/17 


(5:00 p.) 


7 


10/24 


(5:00 p.) 7 


11 


9/17 


(8:00 p.) 5 


10/1 


(8:00 p.) 


5 


10/8 


(8:00 p.) 


5 


10/15 


(8:00 p.) 5 


12 


9/24 


(6:00 p.) 5 


10/1 


(6:00 p.) 


5 


10/8 


(6:00 p.) 


5 


10/15 


(6:00 p.) 5 


13 


10/8 
10/13 


(10:00 p.) 5 
(1:30 p.) 3, 






























► ' 


i 


14 « 


9/29 


(10:30 p.) 3' 


. 9/30 


(10:30 p.) 4 


10/1 


(10:30 p.) 


5 


10/2 


(10:30 p.) 6 


15 

r 


9/24 


(10:30 p.) 5 


10/1 


(10:30 p.) 5 , 


10/3 


(10:30 p.) 7 


10/5 


(10:30 p.) 2 


16 


12/19 
12/21 


(11:00 p.) 7 
(6;30 a.) 2 


12/26 (11:00 a.) 
12/28 (6:30 a.) 


7 
2 


1/2/82 (11:00 a.) 
1/4/82 (6:30 a.) 


7 
2 


1/9/82 (11:00 a.) 7 
1/11/82 (6:30 a.) 2 
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Although some of the stations are showing the programs during- evening 
hours, the majority are showing them during non-prime hours. It appears 
that the stations did not consider the MCT material to be of sufficient 
interest to warrant prime time broadcasts. 

Two of the stations are providing a follow-up session. 

Comments concerning the length of the programs and, the series 
primarily noted that it was too long. Most stations (and this was also 
mentioned in the telephone calls) said that 30-minute segments are easier 
to schedule than 60-minute segments. / 

Comments concerning the production quality of the progifrasrs were also 
cpllected. For the documentary, only one station replied "average" while 
the rest stated "good" to "excellent." Comments were made that the 
content could have been condensed and that the documentary seemd biased 
toward the con viewpoint. 

For the edited tapes, the comments weren't as 'positive, a lthoug h six 
stations responded "fine" to "very good" and four said they hadn't seen 
the tapes yet. Comments ranged from "ho-hum, " "uninspired and 

uninspiring" and "not as good as Advocates , " to constructive comments 

I 

such as "many edits," "too much data" and "content could have been 

% 

condensed." 

The stations were also asked to comment on the production quality, 
information value, and public interest of the shows compared to other- 
shows the stations air. The data from this question are summarized in 

j 

Table 12. 
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TABLE 12 

Quality of MCT Series Compared to Other PBS Shows 

N 



Bottom 
Quarter 



Second 
Quarter 



Third 
Quarter 



Top 

Quarter 



Ppcuaeotary 

Production quality 

Information value 

Public interest/ 
marketability 

Three hearing shovs 

Production quality 

Information value 

Public interest/ 
marketability 



1 

3 



3 
2 



11 

7 



5 
A 



2 



9 
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It appears that the shows were successful in production quality and 
informational value, but doubtful public interest caused them to be aired 
at non-prime times or not at all. Widespread broadcast and exposure for 
the Clarification Process to the general public was not obtained. 
General access to the videotapes will be facilitated by the PBS 
broadcasts. A more effective approach for reaching target audiences 
might be to provide a set of tapes or materials to each State Education 
Department and to provide dissemination svqpport. to encourage and promote 
the use of the materials. This proposal would require that the states 
have the technical capabilities needed to maintain, and distribute the 
tapes. ^ . 

The' Clarification Process is a viable approach given its objectives 
and the outcomes. It met the major objectives cited in Chapter I feut 
other potentially more effective and less costly variations should also 
be considered. This is discussed more fully in the dissemination section 
of the Process Findings and in the final chapter. Specific* items to be 
considered are (a) minimizing the focus on Hhe adversarial nature, 
possibly using a legislative hearing approach in which fewer witnesses 
might provide broader, more in-depth testimony and in which the audience 
might be able to forward questions; (b) reducing the length of edited » 
videotapes from three to one or two hourp; (c) placing more emphasis on 
the written materials to enhance the policy or implementation value of 
the information and (d) developing a stronger dissemination effort 
directed at promoting and supporting use of the materials by the intended 
audiences. - 
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CHAPTER V 
CONCLUSIONS AND RECOMMENDATIONS 

This chapter is organized around the five major evaluation questions 
which are contained in Table 2. The viability question is addressed 
first with responses to other questions viewed as support and , 
elaborations to it. 

I What is the viability of this approach for other 
>>NIE efforts? 

• " fr<- ^ : : 

^he question of viability of the Clarification Process or variations 
of it for future efforts by NIE or others might be characterized as a 
summative question. Based upon results of information collected for this 
evaluation, the Clarification Process appears to be a viable approach for 
providing information to audiences concerned with major education or 
policy issues. * - 

Clarification Process Characteristics . In,, recommending the 
Clarification Process as a viable approach, it is critical to consider 
what constitutes the Clarification Process. Aspects of the judicial or 
adversarial evaluation approach includes direct and cross-examination, 
use of personal judgments and testimony as well as quantitative data and 
is characterized by stages in a case development process. These 
characteristics are intended to produce a variety of- information which is 
more comprehensive than traditional evaluation approaches. However, the 
-'Clarification Process is distinguished from traditional evaluation 
approaches in another important dimension. Specifically, the use of 
videotapes to present personal testimony is a major dimension on which 
the Clarification Process differs from traditional evaluation 
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approaches. In fact/ it is not possible to disentangle the effects of 
using videotapes as a primary reporting and dissemination technique from 
the effects of using the adversarial approach in the Clarification 
Process. Future studies might attempt to look at these factors. One way 
might be to use written documents from a Clatif ication Process or 
adversarial approach as a primary means for communicating the evaluation 
information, and compare this to the more traditional written evaluation 1 
report. Another variation would be to present the results and findings 
from a more traditional evaluation re P°^^ b y Videotape. Finally, fewer 
witnesses testifying in greater detail might increase audience 
participation, decrease 1*||flkdversarial nature which did not appear t 
strengthen the .information gained and decrease the redundancy of 
testimony. 

Given these caveats, information from this evaluation suppoi 

..are^s_of the process.. and pjrpyMes_P^ u tions which might be help! 

future applications. 

/ Audiences found the videotapes to be quite valuable in generating 

discussion and identifying issues related to MCT. They felt that the 

videotapes were much better than written evaluations. "Better" was 

interpreted as more interesting. Additionally, participants or potential 

users felt that the materials were more useful for policy or general 

information purposes and less useful for program implementation or 

revision purposes. Thus, responses to the utility of the Clarification 

Process are consistent with findings of Worthen and Rogers (1977) and 
» 

Wolf (1979) in which adversarial approaches were more useful for 
summative than formative* type decisions. 
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Respondents indicated that areas such as bilingual education, school 
finance, tax tuition credits, curriculum issues, special education and 
mainstreaming and competency based education are issues which might be 
addressed by the Clarification Process. Other topics are listed in 
Chapter 4. 

Timeliness . Given a recommendation that the Clarification Process is 
a viable approach, it is important to offer comments separately for the 
edited videotapes, written materials and the hearing. First, those who 
participated in the sessions conducted for this evaluation provided 
positive reactions that the videotapes were aii interesting information 
source. However, it is also notable and significant that 12 states were 
contacted to obtain a sample of six who were interested in participating 
in the sessions. While it is possible that factors other than their 
interest in the materials affected the decision not to participate in 
these sessions , the expressed reason in each state except one was the 
"untimeliness" of the information. Specifically, five states declining 
either, (a) felt that they had already addressed the issues related to 
MCT, or (b) did not wish to use the Clarification Process materials 
because they might raise sensitive issues given the current political or 
policy factors within the state. < * 

It will generally be necessary to have a movement such as MCT 
somewhat underway before sufficient information" will be available for a 
study. On the other hand, if the study is conducted after most decisions 
and policies have been implemented, then it is less likely that the study 
can have an impact. MCTs have been established in approximately 38 
states, and it is likely that other states 7 have considered MCT. Thus, 
much information exists related to the MCT programs and many MCT 
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decisions and policies are in place. . Given this .factor and the evidence 
that the Clarification Process information is more useful for general 
information in making policy level decisions, it appears that materials 
from the Clarification Proces^would be most useful in reviewing policy 
and informing audiences about MCT and its possible ramifications* 

Use of Materials . Although the Clarification Process materials 
appear to assist in highlighting issues which might need to be addressed 
N in implementing or revising an MCT program, they do not provide specifics 
on MCT implementation type decisions* These might include how to 
maintain student records in MCT programs, how to ensure test security, 
how to handle reciprocity of MCTs across school districts, steps to take 
in developing MCTs V and setting standards* The User's Guide will be more 
useful than the videotapes in this area. Thus, the combination of Uaer 's 
Guide and videotapes appear to have value even in states or areas where 
MCT programs are established. > This use ranges from reviewing current 
policy or implementation decisions to identifying. issues which have not 
been adequately addressed to simply informing audiences, e*g*, a school 
board, about the range of issues in MCT. 

The Hearing . The hearing which was an integral part c-f the 
Clarification Process was not well attended. Several reasons were cited 
in Chapters 3 and 4. The strongest explanation appears to be the fact 
that individuals are not able or interested in committing three days to 
attending a heading. Rather, the information in the shorter edited 
videotapes and User's Guide appear to have more interest* Given that 
individuals who attended the hearing and viewed the videotapes felt that 
soma^additional information was attained in the hearings, future 
applications should maintain some interest in the audience for the 
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hearing. Interest in attending a hearing will be greater if the hearing 
is less than three days, i.e., a one- or at most two-day hearing. 
Attendance at a hearing might increase if earlier invitations were sent v 
or if regional hearings were held in major population areas, e.g., Wolf's 
handicapped hearings. 

Finally, the actual hearing might be critical for state or local 
applications of the Clarification Process in which local audiences were 
the primary focus. For a national study such as the clarification 
Process, it is possible that the hearing can simply be the vehicle to get 
the edited videotapes. Given these factors, the Nlfi/should not maintain 
the hearing audience as a major target for the outcomes of future 
Clarification Process applications without implementing a strategy to 
increase interest and attendance. These strategies could include 
(a) sponsoring regional meetings or hearings and (b) opening the hearing 
to inquiries or responses from the audience to increase the participation 

and potential pay 'off for any attending the hearings. 

/ 
/ 

Did the information presented add to the current 
understanding and knowledge of Minimum Competency 
Testing? 

Results from this evaluation provided evidence that individuals 
gained awareness-type knowledge as a result of attending the hearing and 
viewing the edited videotapes. The knowledge gained can be characterized 
as either reinforcement of existing ideas or identification of new issues 
to consider. Numerous examples of misinformation were cited by viewers, 
although there was no consensus as to specific items. It appears the 
Clarification Process is best described as having synthesized existing 
information which is helpfji^ in understanding and discussing MCT. As 
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elaborated earlier, knowledge from the pro team's case was more easily 
and directly identified with the throe MCT issues debated in the 
Clarification Process. Other evaluation approaches will likely be as 
efficient if the objectiv^ is to generate new knowledge about an area 
rather than to summarize or highlight existing information. 



Did the audiences perceive the information to be 
useful in terms of^pending policy or program 
decisions? 



Opinions did not appear to change pro or con as a result of viewing 
the edited videotapes. However, it appeared that those initially neutral 
toward MCT had moved toward either pro or con positions following the 
hearings but not in the state visits. 

Audiences clearly felt the information .from the Clarification Process 
was more useful for general information or discussion than for 
implementation or program revision decisions. The User's Guide was more 
useful than the videotapes for the latter. General interest in using the 
information was expressed. However, it is important to note that those 
participating in the hearing, state visits or PBS survey represent the 
most interested in using the information. 

It appears the three one-hour edited videotapes are somewhat 
redundant and, in any event, too long to be used- with SEA or LEA 
audiences. Several participants suggested that a one-hour tape 
summarizing the key points would be more useful. A one- or at most 
two-hour tape will be more helpful in training sessions. The reduced 
time in viewing edited videotapes can be used more productively to review 
materials in the User's Guide and to facilitate discussions and 
interactions among the audience in the session. In summary, the 
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audiences felt (a) the information from the .videotapes was useful, 

* 

(b) three hours of videotapes were not needed and (c) the User's Guide is 
a valuable addition to the videotapes. 

Finally, it is recommended that a greater proportion of attention or 
resources be provided to promoting use and dissemination of shorter 
videotapes and a User's Guide to capitalize on the potential benefits. 
If few individuals or agencies actually use the materials, it will not 
much matter that they would have liked them if they had used them. 
Recommendations included providing the materials directly to State 
Education Departments and providing training support for disseminating 
those materials. 



Did the information presented represent a fair 
diversity of viewpoints on each issue that was 
seen as illuminating and clarifying? 

The audiences felt that a variety of viewpoints was presented during 
the hearings, and 'that a fair debate of the issues was provided. Thus, 
the proposed advantage of the Clarification Process to provide a 
diversity of viewpoints which fairly represents both sides .of an' issue 
was accomplished. Although individuals felt that both cases were 
presented clearly, the pro team's case was rated slightly stronger than 
the con team's in clarity of presentations As*<Jiscussed under 
information learned, this is partly attributable to. differences in 
arguing for or against an issue as well as any factors related to the 
team's effectiveness. Finally, it appears based on experiences from the 
Clarification Process and other applications of the adversarial approach, 
that pro and con sides of issues wiaj. generally rest on different 
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assumptions and argumonto or at loaot differing values assigned to 
these. These differences aro likely to leave viewero or an audience with 
the impression that slightly different issues were ppsed by the two 
sides. This appears to bo a natural characteristic which will bo 
associated with pro and con-type arguments. 

A recommendation for the case development stages of the Clarification 
Process is to use the experiences from the Clarification Process in 
assisting to focus arguments around issues and to anticipate that 
arguments will not be perfectly "responsive t6 each other." Again, 
presenting information through individuals testifying vs. through written 
documentation will probably affect the degree to which the arguments are 
perceived to be directly responsive to each other. That is, it is easier 
to lay out arguments in a point-counterpoint fashion when communicating 
in written form than in a hearing setting. 

Audiences also felt that comprehensive cases were provided by the pro 
and con teams, with th^pr^team f s case viewed as more comprehensive. 
Responses as to whether important data were presented or omitted, or 
whether misinformation was conveyed also support that the Clarification 
Process resulted in comprehensive information at the policy or general 
information level. 

♦As specified earlier, gaining information related to actual program 
implementation issues and more discrete level decisions were not cited as 
strengths of the^ Clarification Process. The focus was on the more 
general questions rather than the specifics. One can hypothesize that 
issues framed on operational aspects of tfCT programs might not be 
clarified by the Clarification Process. An example is, "Should alternate 
forms of an MCT be developed to facilitate test security and repeated \ 
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assessments of students." The Clarification Process lo beat suited for 
tho larger issues and moro traditional approaches will probably be at 
loaat an well auitod for tho operational , formative type decisions. In 
the quoetion on altornate test forma , it might be effective to simply 
gather information and present advantages and disadvantages which would 
need to be conoidered in deciding whether to use multiple teot forjns. 
Given tho advantages and disadvantages, it is likely that a decision 
could be made. 



How appropriate were the format and structure of 

the Clarification Process in presenting MCT 

issues, i.e., -in what ways did tho Clarification 

Process help or hinder tho presenting of MCT 

issues? j 

— : — \ 

Respondents in this evaluation valued as interesting the use of 
individual testimony and personal judgment in presenting MCT issues. 
This aspect of the hearings and videotapes is a strength. The use of 
individual testimony is not necessarily a characteristic associated with 
the Clarification Process or adversarial approaches. It might be equally 
possible to present evaluation data using other approaches by 
incorporating video presentations. 

Although the process of using direct and cross-examination was rated 
to enhance the presentation of issues, comments suggested that ' 
cross-examination was not particularly effective in some cases. Since no 
advantages were cited for the attorneys who conducted direct and * 
cross-examination, it does not appear that legal training necessarily 
increases the effectiveness of direct or cross-examination. This 

statement is made in light of the fact that several persons at the 

I 
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hearing and in state visits cited that the Clarification Process 
participants were not experienced in direct and cross-examination. At 
the same time, there was no indication that the participating attorneys 
were more effective than the nonattorneys. , 

A dry run or practice session should be given prior to the hearings. 
This "dress rehearsal" would sharpen the direct and cross-examination and 
is likely to be as effective as any legal background or training. > 
Process Recommendations 

The following conclusions and recommendations are offered In light of 
the strengths and weaknesses of the Clarification Process addressed 
above. These are offered to faciliate future applications or 
considerations of the Clarification Process. The following are specific 
issues which will likely arise and decisions which will need to be made 
in conducting a clarification type process. 

Advisory group . It is recommended that a project involve an advisory 
or constituency group in designing the study, recommending participants 
and suggesting issues. The model used by the NIE in the Clarification 
Process was to place great importance on the advisory group in this 
area. This^f a^ilita^Lsv^p6ft f or and modifications to the study. 

Timelinfes . The timelines for the Clarification Process from the 
initial team meetings to the hearings was approximately 10 months. This 
seems a reasonable time given the national scope of the Clarification 
Process. However, it is important to note that planning and initiating 
the study also covered approximately 10 months. The level of effort 
needed for the Clarification Process was greater than anticipated. 
Decreasing burdens by using information gained in managing the 
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Clarification Process, restricting and coordinating data collection 
across teams and reducing hearing days will minimize the level of effort 
needs. It is likely that in future applications the scope, i.e., 
national or local, and focus of the. study will have more impact on the 
timelines and level of effort needed to conduct the study than whether a 
Clarification Process or other approach is adcpted. \ 

Participant selection . The procedures used to select the team\ 
leaders and hearing officer (s) resulted in nationally known individuals 
serving in the roles of team leaders and hearing officer. It is 
difficult to estimate the criticalness of national recognition and 
acceptance for these roles* However, recognition is likely to be more 
critical in a national study such as the Clarification Process than in 
studies conducted by state or, local agencies. Thus, the availability of 
individuals such as those used in the Clarification Process is probably 
more crucial in national studies than in local or state studies. 

In future applications, the Clarification Process might better be 
managed by the hearing officer rather than a third party contractor if 
the .hearing officer has the time and resources available to perform this 
role. An example of this role was the Northwest Regional Educational 
Laboratory's role in the Hawaii 3 on 2 application. If the sponsoring 
agency, in- this case the NIE, felt it was appropriate and had resources 
to manage the Clarification Process, this role could be served by the 
sponsoring agency in future studies. Again, and it is important to note 
that in the Clarification Process the NIE intentionally did not take; a 
major decision-making role given that MCT is primarily the jurisdiction - 
of state and local agencies. Thus, NIE intended for the Clarification 
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Process to operate fairly independently. It was clear that management 
was a key rol§ in both logistics and decisions required throughout the 
Clarification Process. It is possible that a state agency or local 
agency might desire and be able to manage a study in future applications. 

Composition of teams . The pro and con teams differed in composition* 
and structure, primarily as a result of the type of individuals on the 
teams and the proximity of the team members to each other. Our 
conclusions were that the problems of logistics and communication for the 
con team were greater than those for the pro team. The potential benefit 
for this constraint was wider representation on the con team and greater 
distribution of case development activities across team members. Other 
than the relatively greater difficulty in communication for the con team, 
it did not appear that the team differences clearly affected the quality 

*" ■0"** j ' t ' .'"*■. „ 

or scope of the cases subsequently developed. The pro team was rated to 
have a clearer, more comprehensive case. Again, it is difficult to 
separate the effect of the composition of teams from the effect of the 
assignment to pro or con issues. It is likely that slightly different 
cases will be developed for pro and con issues and that these differences 
effect needs for different teamcompositions. 

Case development stages . The stages used in developing the cases for 
the Clarification Process generally appear adequate. A major stage was 
the framing and selecting of issues. Although this will continue to be 
an important stage, it is felt that the Clarification Process or similar 
approaches can become even more efficient through reduced time or 
resources devoted to this stage. The work done in the Clarification 
Process and thoughts provided by the team leaders (Pcpham 1981; Madaus 
1981) will facilitate issue generation in future applications. Again, it 
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might assist if participants realize that the. final pro and cor/ cases are 

likely to address slightly different m subissues and that these differences 

•• ■ * ' ' 7 \ I 

are a natural by-product of the side of the issue being d^bated^ j 
Another finding, is that the tension in the Clarification Process case 
development increased when the. hearings neared and the teams were not j 

■ " f / ' ' ! .■ 

always consistent in meeting deadlines. Meeting deliverables such as I 

position statements, plans for' direct examination and cross-examination 

/ ' | - 

should be followed and given a high priority 7 . This will facilitate j 

/ / . ; i 

developing cases responsive to each other and will minimize tensions j 

. , - ■ / : ■ I 

during the case development. j 

Tensions occurring between teams during the case development appear 
to be a characteristic op the Clarification Process and other adversarial 
approaches (Worthen and Rogers , 1977). The issue of whether some sites 
fit the MCT definition created tensions in this effort. The issue of 
whether certain evidence could be challenged was another example in the 
Hawaii 3 on 2 project. These examples raise questions about the 
objectives to win vs. to clarify. It is clear that the intent of 
participants was to clarify. It is not clear in adversarial approaches 
when the balance of building the strongest cases moves from a positive 
objective to clarify to a distracting objective to win. The basic point 
to be learned is that the adversarial process is likely to result in 
tension between teams. It is probably not possible to eliminate this 
tension. Instead, it is recommended that tension be anticipated and that 
knowing and anticipating the tension will tend to minimize any negative 
effects on participants or the process. 
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Definitions and stipulated agreements * Evidence in this evaluation 
indicates confusion among the hearing and state site audiences concerning 
the definitions and stipulated agreements used in the Clarification 
Process. It is recommended that greater attention be given to explaining 
the definitions. The fact the teams disputed whether some programs fell 
within the MCT definition is evidence of the sensitiveness of clearly 
understanding the definitions. Finally, the stipulated agreements were 
briefly reviewed at the hearing and in the edited tapes. Audiences cited 
the handicapped testimony as an example of a potential .stipulated 
agreement which was addressed in the hearing. It was felt by some, that 
MCT with handicapped students could have been a stipulated agreement. 
However, it was also expressed that the testimony on handicapped children 
served to more' clearly explain the pro and con teams 1 positions than did 
the stipulated agreements on other issues. If it is not desired to use 
direct or cross-examination ip, introducing stipulated agreements, it is 
important to review the reasons\the teams have agreed to the stipulated 
agreements. \ ^ 

Arrangements for the hearings . Logistics and details for the hearing 
appeared to be satisfactory in most cases. ' An exception was the 
attention given to the audience at the hearing. The audience at the 
hearing was felt by NIE to be an important component of the Clarification 
Process, even though the ultimate products were the videotapes. A 
shortened hearing of one or two days with earlier invitations t6 attend 
the hearing "will facilitate greater participation. Additionally, 
regional hearings could be sponsored, if audience participation is 
important. Finally, in a national study such as the MCT Clarification 
Process, the importance of an audience at the hearing could be viewed as 
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minimal with greater attention given to 1 disseminating the products from~^ 
the hearing. 

NIE has widely disseminated information about the videotapes and 
materials from the Clarification Process. Other efforts might maximize 
the impact and influence of the materials from the Clarification 
Process. In future applications it would be desirable to have (a) a 
training package consisting of a one- or two-hour edited videotape, 
written materials and trainer's guide, and (b) a dissemination plan in 
which resources would be devoted to setting and providing technical, 
assistance-type sessions to intended audiences. 

The Clarification Process findings suggest that even extensive 
information from a three-day hearing can probably be edited into a one- 
or two-hour tape. Additionally, a one- or two-hour tap^can^fapd^litate 
interactions and will allow more time for interactions. A one-day 
training or dissemination session is suggested due to the interest, 
logistics and economic restraints found in longer sessions. 

The decision to disseminate materials through the PBS stations 
appeared to be effective in the sense that wide awareness was generated 
of the Clarification Process. However, there does not appear to be 
wide-spread "interest in using three hours of information on MCT. If PBS 
broadcasts were to be considered for dissemination in future 
applications, it is recommended that at most a one-hour . show be developed 
as a product of the process. Additionally, the decision to disseminate 
the Clarification Process' edited tapes through the PBS stations affected 
the perceptions of the Clarification Process participants. These effects 
included types of witnesses used, cases argued and logistics, concerned 
with taping these shows. While the influence of PBS might be argued by 
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some, it is likely that Sidney Poitier and Ralph Nader would not have 
been proposed as witnesses had not these shows been planned for broadcast 
on PB^ stations. Although it is uncertain whether the impact of the PBS 
broadcast materially affected the final cases presented by the teams, it 
is clear that these broadcasts contributed to the tensions and concerns 
felt by the participants in the Clarification Process. Disseminating 
videotapes and materials directly to State Education Agencies might be 
more effective and efficient than through PBS stations. 
Conclusions 

This evaluation reports addressed the process and outcomes from the 

i 

MOT Clarification Process. It appears that the Clarification Process is 
a viable approach for studying major education or policy issues. The 
information produced from this application appears to. be most useful for 
summative or general information purposes. Recommendations based upon 
experiences from the Clarification Process will be helpful in future 
applications. 

It is also recommended that future studies or efforts attempt, to 
Begin addressing the extent to which the adversarial nature of the 
Clarification Process, rather than the use of individual's testimony and 
presentation through videotapes, contributed to the positive evaluations 
of the Clarification Process. It was not clear in the study that the 
positive outcomes were dependent upon the adversarial aspect of the 
Clarification Process or more a function of the videotapes and 
individuals' testimony. Efforts for studies in this area will enhance 
evaluation utilization and impact* /• 

i ' y \ 

\ \ t 
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Finally, it is likely that reduced days for a hearing, shorter edited 
tapes, fewer witnesses presenting more in-depth testimony and more direct 
dissemination to a narrower audience will result in a less costly effort 
with similar or greater impact. 
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Appendix A 



Questionnaire Distributed 
at 

MCT Clarification Process Hearing 
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11. U your opinion ttit «ri tin tut piriuiilft points udi 
a* pro Utt ^^^^^^ 



bi con Uu 



H 

00 



1], Oftrslli *tot do you think art tbs stcsngttas of hsarlngs? 



Ki Ofirill, vhit do you think in thi vukNUii of thi hiitlngi? 



Hearing Evaluation Form 



Tin Dopsrtaent of Education*! national Inetititi ol Education haa 
Instituted a Mjor national ptojact alied at clarifying aota of thi 
■oat lallmt laaoaa concerning Hlnliui Coapattncy Taatlng |HC?| and 
its potantlal affacti .poaltlvi or negatlvi, on atudanta and tha 
quality of aducatlon pfogrui; Tbla ptojact li designed tot - 

1. (ncourego tha preeentatlon of diverse viewpoints and 
differing Interpretation! of Nlnliua Cooipetency leeting 
(HCT) policy end progreu In en orgeniied and public lennar; 

2. conelder the perception of those who are or Will be 
effected by the exieting or uerglng pollcyi end 

]. prowte a fair end reeeoned debete of the laaoaa. 

i - c 

lour perception of the effectiveneae of the Clarification 
Bearing is needed not only to evaluate thte particular 
application but also any potential' eppllcetione in the future. 
Pleeie reepond to the encloeed questions trhlch will be collected 
et the door ee you leeve for the day. 



Any ottur ooannti nith spKlflo ivuplu iri *1cqm md Midtd, FmI 
trtt to prorldi th* dlrictly blow or by contesting Oiry D, titn, 
NorthMlt Riglonil Iducitlonil Uboritory, 300 S.it. Sixth Annul, 

toftiind, Of^on am, i (loo) nmjm.' 
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filet other dsylsl hen you sttinM the nestings? 

□ JULVB □ JULY9 



) 



Which typt of sgtncy 4o you > 

! 

Q FEOBtAL D STATE D LOCAL Q NATIONAL ORGANIZATION D OTHER 



aWhidh oni of thi following then cittgorlu bat npriimti your 
roll or ths roll of Individuals In your orgintutlonr 



D BlffignslbU foe potlclii p mponilbli for thi 

•bout vhtthtc i »lnli« lipleimtitlon of in 

owpitincy •tilting (HCt| HCT progni (Exupliii 

piogm should t» ichool evhlnlittitosri, 

lipliiinttd (Exupliit tuchiri) 
ichool board sasb«rs» 

ititi Itylilitoti) ' 



L 



O Pltsctlyitfictid by in 
HCT progrii JExiiplMi 
pinnti, iptclil 
tntirtit gtoupil . M 
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Direction* i Plaoa aa [T\ in tba ben af tat ««ch itaa which bat 
mpraaants your cmpoom. 



t , w 

t X. Dow would you rata your opinion of Mlnlaua Qoapataacy ^Z* ' 

■*T mating <HCT) bafora tbaaa baarlaga? U U U LJ LJ 

| now would too rata tout opinion of ICT a^twr tba baarinoa? □ □ □ □ □ 

r V— v ttmmh 9mm um« ©# * 

MtMii MfiMit Knowiaat* bw*iw# 

X. a. lav would you rata your kaovladsa of KT bafora tha n n n n 

5 baaxlofa? LJ LJ LJ LJ 

"J Ma Hi 

b. Did you 9ala any now knovladga froa tha baarinf? 

□ 

ftriafly llati 



MWllV ) 

9mm Mow Q*9*m 



□ 



? J. tba baarlnga prorldad a fair dabata of tba laauao. *TT A ^T ^^T** °*^T M 

7 □ □ □ □ 

» • 

k 4. Tfca proeaaa of ualng taatlaony and croaa-asaalnatlon □ □ □ □ 

aahnaoad tha praaantatlon of laauaa? . 

5. Tba two taaka praaantad; a ooaprahanalva oaaa la aupport of 

; tbair poaltlona? 



a* pro taai □ □ □ □ 

b. oon taaai » □ □ □ □ 

<• A r gi a ia n ta wara praaantad elaarly byi 

a. pro tana < □ □ □□ 

oam baa. ^ □ □ □ □ 

7. tba ana of individuals* aubjactlva judgaanta annaaead rn r-i r-i r-i 

.tba praaantatlon of laauaa. u u u u 

•• a auriaty of Ttawpolnta waa praaantad during tha baarlaga? □ □ □ □ 

9. Vara tha Boat iaportant MCT laaoaa addraaaad la tha 4 *• 

baarlaga? . □ □ 

Want otbara. If any, alght barn baaa aora iaportant? 



(. Vocy Uftla or No 

f 10. would you ba intaraatad in ualng tha thraa ana-hour adltad "TIT** - "«**••«*■ iwooi 
tapaa f com tha baarlnga? □ □ □ 

i UU lata tha una of iaforaatlon f ton tha haarlnga fort % UMru i ^'HL^,** 



*j a. foraulatiag policy about adopting aa ICT prograa □ D □ 

' b. aaaiating with taplanantlng an MCT prograa Q Q Q 

£ o. laformlag tha oaaaral public about MCT Q □ " □ 

£ a\ othar, plaaaa apaolfy □ □ □ 
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Letter Sent to State Departments 
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Northwest 

Regional ^ ^ 

LL^JlLL^ 1 11 h\ 300 S,W, Sixth Avenue • Portland. Oregon 97204 • Telephone (503) 248-6800 



September 25, 1981 



Dr. Thomas Fisher ^ 
Student Assessment 
580 Knott Building 
Tallahassee, Florida 32301 

Dear Thomas: 



The Northwest Regional Educational Laboratory is currently involved in 
evaluating the videotapes and written materials produced from the NIE 
CLARIFICATION Hearings on minimum competency testing. Part of that 
evaluation involves meeting with groups of practitioners and policy 
makers in various states and collecting their views on the usefulness of 
the videotapes. After talking with your office on Thursday, I decided itT 
might help to send you some more information about the meeting we propose. 

A possible agenda for the meeting is listed below. The agenda is 
flexible, and it might be desirable to adjust the agenda. For example, 
if most of the participants view the PBS broadcasts, it might be 
desirable to view all the tapes and then discuss/evaluate the materials. 
However, I assume that it is likely that many will not have viewed the 
tapes. 



Morning 

I. Review of Clarification Process 9* 00 - 9:20 

II. • Introduction to Purpose of the Session 9:20 - 9:30 

III. Viewing of the First Videotape 9:30 - 10:30 
Break \^ 

IV. Discussion of Appropriate Section of 

User's Guide 10s40 " 11:00 

V. Collection of Participant Reactions 11:00 - 11:45 



Lunch 



11:45 - 1:00 
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AN EQUAL OPPORTUNITY EMPLOYER 



Dr. Thomas Fisher 
September 25, 1981 
Page 2 



Afternoon 

VI. Viewing of the Second ^fdeotape 1:00 - 2:00^ 

VII. Discussion of Appropriate Section of.. 

User's Guide 2:00 - 2:10 

VIII. Collection of Participant Reactions 2:10 - 2:40 
Break 

IX. Viewing of Third Videotape 2:50 - 3:50 

X. Discussion of Appropriate Sections of 

User's Guide 3:50 - 4:00 

XI. Collection of Participant Reactions 4:00 - 4:30 

It is also helpful to outline- the audiences which we think might provide 
.input related to the information and format of the Clarification Process 
videotapes and materials. We are flexible and want to work with you to 
be sure that the group (s) will meet your needs and priorities as well as 
ours. 



Participants ' Approximate Numbers 

SEA Representatives 

Chief State School Officer or Representative 1 

State Testing Director \ 1 

School Board Member \ , 2 

\ 

LEA Representatives (large and small districts) 

Superintendent 3 

Testing Coordinator 3 

Curriculum Specialist 2 



\ 



\ 



/ 



\ 

\ 
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Dr. Thomas Fisher 
September 25 , 1981 
Page 3 



participants 



Approximate Numbers 



Teacher (representative of teachers' 
organizations) 



3 



School Board Member 7 



2 



Legislative Representative 

State Legislator or representative such as 
legislative analyst 



2 



Special Interest Groups 1 Representatives 



Specific groups particularly important with state 



3 



Our plans are to collect data in states the last week of October and 
first week of November. We will be happy to work out a date which will 
be convenient to you. f \ 



I look forward to talking with you about our project. Thank you for your 
time. 

Sincerely, 
Randy Demaline 

Evaluation of MCT Clarification Process 
RDrpsp 




j ■ 
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Questionnaire Distributed 
to 

PBS Stations 
Concerning 
"Who's Keeping Score 11 Series 
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J. Ion iany of thou Institutions, to your knwiidgt ricotdtd thi 
prograst or regueitid tapei of thi probata for litor ml 



JSBSL 



MootdtJ Related Tipti 



School Dlitrlcti 

i - 

Coseunlty/Junlor coll. 9,1 

rour-ywr oollegei/unlvtnltlei 

Othiri (pluM ipKlfy) 



e\ Wilt li your opinion of thi hiitlnga fomit to clerlfy or intor* 
audlencei ibout ujor educational lasues? 



7, f « ilillir Uarirwi tpptoich li mid to iddrm other tducitloml 
Issues, vhit liprotmnti wuld yoo suggest for thi production of 
iirlu bind on thi proceedings? 



PUBLIC TELEVISION STATIONS 
EVALUATION FORM FOR 

WHO'S KEEPING SCORE? 
Minimum Competency Testing Television Series 



Thi Depertoent of educetion's Mitionil Initltuti of eduction hie 
instituted . „]or mtlonol project .| H d it ei.tUying .o,7o[ "hi 
»ost nllint Issues concerning Hlnlnus Coipitincy Totting |NCt| mi 

fi ,Bk m ,H ! ct I p0,ltlw ot M ' atlw ' on curricula, 
•nd the quality of iducitlon pcogrus. Thli piojict li designed toi 

1. incounge thi priimtitlon of dlviru vieupolnti and 
' differing Intirpntitloni of Nlniiw Conpitmcy Tilting 
IHCT) policy ind program In in orgmliid ind public iinniri 

1. oomldir tin pirciptioni of thoii who in or will be 
iffictid by thi ulitlng ot Merging pollcyi «nd 



prcsote 1 dlt ind nuontd debate of thi liiues, 



four perception of thi iffictlnniii of thi television series on 
Klniiui Ccspetency Tilting ,|HCT1 li nudid not only to ivtluiti 
thli application of television but 1I10 my potential 
applications In the future, One (on should be cupleted (or 
your itatlon, Pleaie respond to thi encloiid queatlons and 
return In the solt-eddressed envelope by October 1], 1991. 



0. Pleats provide any other oonsnti you wish to lake about HHO's 
UEPING 8CORB, 



raraon responding, Title" 



Statlon/Netvotk 



Pleasi return toi Rete Richardson 
by October J], Del Southern Educational Ccmunlcatlon Association 
P. 0. Boa 59(6 

Coliabla, South Carolina 2)250 



THANK you, 
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1. Did or Hill fow station lit my of thi four to-ilnttti pogrw In o. troduotlon qwllty o( Frogcia 1 |fhi DoouMntaryH 

tht iitlil? 



a, If no, why not 



d, Production quality of Pcogrmi H (Heating Diyi Highlights)! 



b, If y»i, which oni(i) and whin 
prcxjraa 1 |Doct*entary) 
ProgcM 2 (Hearing Day 1) 
Ptogria 3 (Hairing Day ]) 
Progra* ( IHearlng Day i) 



Datl(l) TlMjl) 



c, Has there a local follow-up discussion produced and broadcast? 

Yes • No , 

If yea, aftir which prolans) 



3hov 



Data 



TUi 



Prograa L Docwentary 
Prograa 1 Kiarlng Day 1 
Prograa 3 Hairing Day 2 
Prograa t During Day ) 
Other followupi 



2. Hhit art your opinion! of tht following eltaent(e) of thi ilnl-serlee? 
a. Length of ptograas (60 alnutiaji _. -_ m 



O b. Length of strlee |4 program) \ m 



I How don the HCT ilnl-iirlii cohere to othir ihowl your ititlon hii 
alrid? 

Bottoe Stoond Third Top 

Quarter Quartir Quirtir Quarter 



Docwentary ihowi 

Production quality [] 
information vilui 



Public Intiroit/ 
larkitablllty 



D 
D 



Thru haarlng ihowii 

production quality Q 
intonation value 



Public Intinik/ 
■instability 



D 
D 



0 
D 



D 
D 



D 
D 



D D 



D 
D 
D 



D 
D 

D 

D 
D 

D 



(, Appronliately bow amy of thi following ideational Institutions an 
In your broadcast mviragi aria? 



agency 



Nwber 



School Dlatrlcti , 

Ccwunlty/junlor oolligis , 

four-year colleges/ 

universities — — 

Other (pleass specify) . 
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Questionnaire Distributed 
at 

Site/visits 
for the 
Edited Videotapes 
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Evaluatlnn Par* 
for 

Minimum Cnnipetency Testing Clarification Hearing 
Videotapes 1 

I 

The Department of Education's National Institute of Education ha a 
Instituted a ma |or national project aimed at clarifying some of the 
most aallent Isauea concerning Minimum Competency Testing (MCT) and 
Ita potential effect, pnattlve or negative, | on atudenta and the 
quality of education prog no a. This project Is designed to: 

1. encourage the presentation of diverse viewpoints and differing 
Interpretations of Minimum Competency Testing (MCT) pnllcy and 
programs In an organised and public manner; 

2. cnnslder the perceptions of those who are or will he affected hy 
the existing or emerging policy* and 

1. promote a fair and reaanned debate of the Issues. 

Your perception nf the effectiveness of the Clarification Hearing 
shows Is needed not only to. evaluate thla particular appllcatlnn hut 
also any potential appllcatlnnn In the future. 

Thank you for ynur assistance. 

1. Hnw would you rate ynur 
opinion of Minimum Compete nc 
Testing (MCT) hefnre viewing 
these showings? 

2. Hnw would you rate ynur 
knowledge nf MCT before 
viewing these shnwlngs? 

3. Did you see I'he dnnimentary 
on MCT, "Uhn',9 Keeping 
Scnre", that was aired 
prevlnus tn tUtse shnwlngs? 



y Strongly Siro< _ 

favoi Favof Naulial OPPota Op pot a 

□ □ □ □ □ 



Vaty nnomU Unowl- §oma Lillla ot Ho 

adg.au I a adgaabta Knowladfla Rftuwladfla 



□ □ □ □ 



□ 
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Quest lorn Concerning Show One 



1. Have you seen this show 
before] 

A variety of viewpoints on 
HCT were preaented In this 
show* 

)* This show presented a fair 
discussion of NCT Issuei. 

4> Argents were presented 
clearly by 

i, the pro tea* 
b. the con team , 



5 1 Did you Rain any new 
knowledge about HCT froa this 
show? 





VII 






n 
U 


u 












filHKM Olllftit 


a 


D 


a o 


a 


D 


D D 


D 


a 


D D 


D 


□ 


D Q 








NoNn 




GtlltK 








D' 


D 


a 



6, What, If any, waa the wit Important new knovli , on Rained fnn 
thin show mil how will you use It? 



A, Hai there any Information that you feel wai not uieful In 
clarifying the HCT laaueat 



9, What uere the float persuasive polnte wde hy 
a. the/tiro tenal 



b. the con teai? 



10, Mho were the moot effective wltneaaes for 
a, the pro teai and why? 



J. Was there any information preaented that waa Inconalitent with 
vhat you know about HCT? \^ 



b. the con tean and why? 



ERIC 
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Hi Any general comment about this show? 
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Queetlons Concerning Show Two 



1. Have you seen thle ahov 
before? 

2. A variety of viewpoints on 
HCT were presented In thle 

3. This ahov presented a fair 
discussion of HCT Issues* 



0 4. Arguments were presented 
clearly by 

a. the pro team ' 

b. the con tean 

5. Did you gain any new 

knowledge about HCT from this 
show? 





Mi 


Hi 






D 


D 




Agno 


inn 




iMOflQly 
Dilty'H 


0 

(i 


D 


D 


0 


D 


D 


D ' 


D 


D 


D 


D 


D 


0 


D 


D 


D 


NoNm 




lutilwlillly 

Giiim 
<XA««lldgt 




D 


D 


D 





6. What, If any,, was the nost Important new knowledge you gained from 
this show and how will you ubc It? 



8. Vai there any Information that you feel was not useful in 
clarifying the HCT Issues? 



9. What were the Boat persuasive points lade by 
a. the pro teas? 



b. the con tean? 



10i Uho vera the lost effective witnesses for 
at the pro teaa and why? 



c 



?. Hob there any Information presented that was Inconsistent with 
what you know about HCT? 



b. the con teaa and why? 



ERIC 
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11. Any general comment about this show? 



Question! Concerning Show Three 



li Have you seen this show 
before! 

2*. A vatlety of viewpoints on 



5. Did you gain any new 

knowledge about HCT from t hi 6 
allow? 



Vfl No 

D 0 



8. Has ther| any Intonation that you feel was not useful In 
clarify! g the HCT lasuea? 



lubiltfliilly 
NoNt* SomNiw Gimm 
K/imIrJo* IMidp Ioo*Ik)bi 



a a 



6. Wiat, If any, was the most Important new knowledge you gained fron 
thin show and how will you use It? , 



HCT were presented In this 
ehowi 


D 


D 


DiiiVM 

D 


Oliifliii 

D 






This show presented a fair 
discussion of HCT Issues, 


0 


0 


' Q 


D . 


9i What wsre the noet persu 


i 

aslve points wde by 


Arguments were presented 
clearly by 










a* the pro tean? 


/ 

> ; 

i 

! 

■ 

i 

! 


a* the pro tean 


D 


0 


□ 


D 




. 1 

■ 1 
i 

♦ j 

/ 


b« the con tean 




,0 


0 


D 


b. the con team? 


/ 



! " 

10» l/ho were the most effective witnesses for 
a. the pro tean and why? 



h Was there any infqmatlon presented that was inconsistent with 
what you know about HCT? 



h. the con team and why? 



9 

ERLC 



4n> 



11. Any genernl cnmmont about this bIiou? 

/ - 
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Final Questions about the HCT/CP 



B. What HOT issues are you most concerned about) 



1, How would you rate your 
opinion of HCT after viewing 
all three shows? 

2. The Clarification Process' 
use of Individuals' teat iaony 
and personal Judgments 

( enhanced the presentation of 
the HCT Issues. 

J. The Clarification Process 1 
use of direct and 
croBS-exanlnatlon enhanced 
the presentation of the 
Issues* - 

4, The Clarification Process 
provides an opportunity for a 
fair discussion of the HCT. 
Issues. 

5. Comprehensive cases In 
support of. their position 
were presented by 

a. the pro team 

b. the con tean 



fivof F»vm Ntuldl Opp*M OPpoM 

0 Q 0 0 Q 



SiiOflllf Stilly 



D 



D D 



□ on 



a a . a a 



D D 0.0 

o a d o 



Was there any Important Information that una omitted from the 
shows by either the pro or con teams? 



7 ^ ^ yon feel that there are more Important HCT Issues that were not 
ERJC (,rBeBe( ' ^ 1,18 Bl,ow? 



9i Do you think the lnforaatlon In these shows will be 
useful to you In relation to these issues? How? 



10. Kate the use of the Information from these shows for; 



St formulating a policy 
about adopting an HCT 
program 

b. assisting with the 
Implementation of an NOT 
program 

Ci Informing the general 
public about HCT 

d. revising or dropping an 
HCT program 



HiM „ Hull 
UilM Unlttl Um 



0 
D 

a 

D 



a 

D 

a 
o 



D 
0 
0 
D 



Hi How does the video presentation format compare to well 
written evaluation reports? 



The shows are b etter 
than a written 
evaluation! 

The shows are about the 
same as a written 
. evaluation. 

The shows are not as 
gond as a written 
evaluation. 



a 



D 



7 



Questions about the Clarification Proceie 



li laaed on these Bhows, what do you think are the strengths of the 
Clarification Process? 



H 



2, Baaed on these Bhovs, what do you think are the weaknesses of the 
Clarification Process? 



4 .Questions Concerning the User's Guide 



Vii 



li la the User's Guide useful as a supplement to the shows? 0 
Consent^: 



No 

D 



2 i How would you rate the User's Guide for 



a. formulating s policy 
about adopting an HCT 
' ■ progiw t I 

bi assisting wlttnhe 
Implementation of an 
HCT progran 

Ci Informing the general 
public about HCT 



HI|My 


Uiilvl 


Unit 
Ui» 


D ■ 


D 


• D 


D 


a 


D 


D 


a 


O 



3. Vbat do you like about the User's Guide? 



4* Are there any changes that you would suggest that 
h What other educational Usuea do you think could be covered ««M ^e It aore usable? 

effectively using the Clarification Process? * 
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APPENDIX E 
SITE VISIT DATA SUMMARIES 



144 



1 



APPENDIX E 
TABLE 1A 



Responses to Testimony and Cross-Examination Enhancing Presentation of Issues, 





Strongly Disagree 
(1) 


Disagree 
(2) 

N (I) 


Attee 
(3) 

IM 


Shronfflv Aorpp 

(4) 

LSI 


Number 
RcBponoing 


Mean 


Standard 
Deviation 


HEARINGS 



















3 (5) 


8 (13) 


33 (53) 


10 \17) 




1 ft 

3.0 


n 

.8 


SITES 








■ 








Los Angeles, CA 


1 (6) 


3(17) 


13 (72) 


1 (6) 


18 


2,8 


.6 


San Francisco, CA 

H 




6 (43) 


7 (50) 


1 (7) 


14 


2.6 


.6 


01 Florida 






• 6 (67) 


f . 3 (33) 


9 


3.3 


.5 


Illinois 






9(64) 


5 (36) 


14 


3.4 


.5 


Montana 




1.(4) 


20 (80) 


4(16) 


25 


3.1 


,4 


Texas 




1 (9) 


6 (55) 


4 (36) 


ii 
ii 


Ji J 


♦ D 


Wisconsin 


\ 


4 (19) 


11 (50) 


6 (29) 


21 


3.1 


.7 


Special Interest 














;v 


Groups 


1 (13)- 


4 (50) 


3 (38) 




8 


2.3 


.7 


TOTAL SITES 


2 (2) 


19 (16) 


75 (63) 


24 (20) 


120 


3.0 


.7 



o 

ERIC 



17- 
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! . APPENDIX! 

, TABLE IB 

' ■ i * 

Responses to Personal Judgments Enhancing the Presentation of Issues 

t 

\ 



<• 


Strongly'Dlsagree 
(1) 


Disagree 

(2) 


Agree 
(3) 


Strongly Agree 
(4) 


Number 
Responding 


Mean 


Standard 
Deviation 






LSI 


LSI. 


LSI 








HEARINGS 


2 (3) 

» 


8 (14) 


36 (62) 

»/W 1 Ufa J 


12 (21) 


58 


3.0 


.7 


SITES 

Los Angeles, CA 


1 (6) 


5 (29) 


10 (59) 


1 (6) 


17 


2,6 


,7 


h San Francisco, CA 


2(14) 


9(64) 


3(21) 


14 


3,0 


.6 , 


Florida 




4 (40) 


5(50) 


1 (10) 


10 


2,7 


,7 


Illinois 






8 (57) 


6(43) 


14 


3,4 


,5 


Montana 




1 (4) 


19 (76) 


5(20) 


25 


3,2 


,5 


Texas 


/ , 




8 (73) 


3 (27) 


11 


3,3 


,5 


Wisconsin / 




2 (9) 


13 (57) 


8(35) 


23 


3,3 


• 6 


Special Interest 
Groups 




< 

1(14) 


6 (86) 




7 


2,9 


.4 


TOTAL SITES 


: 1 (1) 


15 (12) 


78 (65) 


27 (22) 


121 


3,1 


,6 



\ 



APPENDIX E 
TABLE 1C 

Responses to a Statement that, Comprehensive Cases Were Presented 





Strongly Disagree 
(1) 


Disagree 
(2) 


Agree 
(3) 


Strongly Agree 
(4) 


it t. " 

Runi-er 
Responding 


Mean 


Standard 
Deviation 




13) 


LI?) 


im 




! 

i 

i 






HEARINGS 










- - , , , 






' Pro Team 


1 (2) 


7 (12) 


31 (53) 


19 (33) 


58 


3,2 


.7 


■ Con Team 


2 (4) 


17 (30'l 




1 • 


56 


2.8 


,7 


SITES: Pro Team 








i 








Los Angeles, CA 

H 




3 (17) 


11 (61) 


< (22) 


18 


3.1 


.6 


m San Francisco, CA 




4 (29) 


7 (50) 


3 (21) 


14 


2,9 


,?• 


Florida 




1 (10) 


6 (69) 


3 (30) 


10 


3.2 


.6 


Illinois 




1 (7) 


8 (57) 


5 (36) 




n « 


,0 


Montana 






iO 


1 /till 

J (13) 


24 


2,9 


.6 


Texas 






8 (73) 


3(27} 


11 


3,3 


.5 


Wisconsin 






15 (65) 


8 (35) 


23 


3.3 


,5 


Special Interest 
















Groups 




2 (25) 


6 (75) 




8 


2,8 


.5 


TOTAL 




16 (13) 


77. (63) 


29 (24) 
.. i 


122 


3.1 


.6 
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TABLE 1C (Cont.) 

Responses to a Statement that Comprehensive Cases Here Presented 





Strongly Disagree 


Disagree 

w 


Agree 

(3) . 


Strongly Agree 
(4) 


Nuiber 
Responding 


Mean 


Standard 
Deviation 




am 




UI) 










SITES : Con Team 














\ 
\ 


Los Angeles, CA 




n (6i) 


6(33) 


I (6) 


18 


2,4 


.6 


San Francisco, CA 




6(50) 


5(42) 


I (8) 


12 


2.6 


.7. 


Florida 


i in 


2 (20) 


6 (60) 


1(10) 


10 


2,7 


.8 


Illinois 




1 (7) 


10 (71) 


3(21) 


14 


3,1 


.5 


£ Montana 




7 (29) 


13 (54) 


4(17) 


24 


2.9 


.7 


\ 

Texas 

\ 




1 (9) 


7 (64) 


3 (27) 


11 


3,2 


.6 


\ 

Wiflconsin 


1 (5) 


7 (32) 


U (50) 


3(14) 


22 


2,7 


.8 


Special Interest 
Croups 




6(75) 


2 (25) 




8 


2,3 


,5 


TOTAL 


2 i'2) 


41 (35) 


60 (50) 


16 (13) 


119 


2,8 


,7 
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cj 



APPENDIX E 



TABLE 2B 

Responses to a Statement that the Clarification Procesc Ptt id 
a Fair Discussion of the Issues 





Strongly Disagree 
(1) 


Disagree 

(?.) 


Agree r 
(3) 


Strongly Agree 
(4) 


Number 
Responding 


Mean 


Standard 
Deviation 




LID 


US 


US 


US 








HEARINGS 








1 








i 

i 


2(3) 


8(14) 


38 (64) 


11 (19) 


59 


3.0 


,7 

/' 


CTTPC 

bilKo 












• 




f A a* UnAA I /i/l P A 

LOfl Ang6168| In 


1 t(s\ 




in (f\V\ 


1 f6i 


16 


2,7 


,7 


H 

^ ban rrancisco, ia 




0 ^jj 


\ \J\)) 


1 (7} 
i \i ! 


14 


/2,6 


.6 


r lor Ida 


i 




1 \lv) 


i no) 


10 


2.9 


.6 






1 (7) 


B (57) 


5 (36) 


14 


3.3 


.6 


Montana 




3 (12) 


1? (68) 


5 (20) 


25 


3.1 

1 


.6 


Texas 






6(55) 


5(46) 


if 


! 3.5 


.5 


Wisconsin 




4 (18) 


15 (68) 


3(14) 


22 


: 3.0 


.6 


Special Interest 
Groups 




2 (25) 


4 (50} 


2 (25) 


,8 


i 

3.0 


•8 


TOTAL SITES 


1 (1) 


22 (18) 


74 (62)_ 


23 (19) 

i 


120 


3.0 


.6 



ER?C 18u 
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APPENDIX E 
TABLE 2C 

• Argumenta Presented Clearly by Pro and Con 



- 




Strongly Disagree 
(1) 

LS) 


Disagree 
(2) 

NJ1) 


Agree 
(3) 


Strongly Agree 
(4) 

US 


Number 
Responding 


Mean 


standard 
Deviation 

* 


HEARINGS 


















Pro 




1 (2) 


7(12) 


28 (48) 


22 (38) 


58 


3.2 


,7 


Con 




1 (2) 


18 (32) 


30 (53) 


8 (14) 


57 


2,8 ' 


.7 


SITES 


















Show I 


Pro 


1 (1) 


U (8) 


77 (57) 


46 (34) 


135 


3,2 


.6 




Con 


4 (3) 


21 (16) 


82 (62) 


26 (20) 


133 


3,0 


,7 


Show 2 


Pro 


1 (1) 


11 (8) 


88 (66) 


33 (25) 


133 


3,2 


,6 




Con 


I (1) 


25 (») 


77 (58) 


30 (23) 


133 


3.0 


.7 


Show 3 


Pro 


3 (3) 


5 (4) 


72 (63) 


35 (30) 


115 


3.2 


,6 




Con 


1 (1) 


19 (17) 


70 (61) 


25 (22) 


115 


3,0 


.6 




APPENDIX E 
TABLE 3A 

Audiences' Initial Knowledge and Gain In Knowledge 



HEARINGS 





Little or No 

Knowledge 

(i) 


Some New 
Knowledge 

(2) 


Knowledgeable 
(3) 


Ve,ry 

Knowledgeable 
(4) 


Number 
Responding 


Mean 


Standard 
Deviation 




Hi) 




us 


L® 








Before the Hearing 


A / ft \ 

2 (3) 


15 (24) 


28 (45) 


17 (27) 


62 


3.0 


.8 


Gain in Knowledge 


No New 
Knowledge 
(1) 


Some New 
Knowledge 
(2) 


Substantially 
Greater Knowledge 
(3) 










6(10) 


40 (67) 


14 (23) 


• 


60 


2.1 


.6 


h SITES 

H 

Before the Shows 


Littie or No 

Knowledge 

t\ \ 
U) 

L® 


home New 
Knowledge 
UJ 

Li? 


Knowledgeable 
(3) 

LB) 


Very 

Knowledgeable 
(4) 






« 


Los Angeles, CA 


1 (5) 


2 (10) 


M35) 


10 (50) 


20 


3.3 




San Francisco, CA 




3 (20) 


8 (53) 


4 (27) 


15 


3.1 


,, 


Florida 




1 (9) 


6 (55) 


4 (36) 


11 


3.3 


.6 


Illinois 




5(31) 


8 (50) 


3 (19) 


16. 


2.9 




Montana 


i w 


, 10 (37) 


1 13 (48) 


3(11) 


27 


2,7 




Texas 




2 (14) 


7 (50) 


5(36) 


14 


3.2 




Wisconsin 




5(19) 


11 (42) 


10 (39) 


26 


3.2 




, Special Interest 
Groups 




2 (29) 


4 (57) 


1 (14) 


7 


2.9 




ErIcal 


2 (2) 


30 (22) 


(4 (47) 


40 (29) 


136 , 


3.0 


,e 185 



Jfi 



TABLE 3A (font.) 
Audiences' Initial Knowledge and Gain in Knowledge 



SITES 


• 












Gain in Knowledge 


No New 
Knowledge 

us 


Some New 
Knowledge 

us 


Substantially 

P«*An vnrtl.il adna 

oreaier Miowieage 

LIS) 


Number 

RaanntirHno 

I\GD|JUllUlll|J 


M0 /in 


Standard 

FlotH flf Inn 


Show 1 


43 (32) 


83 (62) 


8 (6) 


134 


u 


.6 


Show 2 


54 (42) 


68 (52) 


8 (6) 


130 


1.6 


V .6 


Show 3 ! 


57 (50) 


57 (50) 


1 0) 


115 


1,5 


,5 



in 



J 



t 



■ 18G 187 
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APPENDIX E ' 
TABLE 4A 

Opinions of MCT Before and After the Clarification Process 



H 

00 



Strongly 
Oppose 

(i) 



mm 

Before U (7) 

After /5 (8) 

SITES 

Lofl Angeles, CA 
Before 

After ' 1 (6) 

San Francisco, CA 
Before 
After 



Oppose 
(2) 



Neutral 
(3) 



2 (10) 
1 (6) 



3 (20) 
2 (13) 



5 

A (22) 



4 (27) 
3 (20) 



Favor 
(4) 



Strongly 
Favor ' 
(5) 



LB L® , UP gjg U9' 



13 (21) 18 (29) 20 (32) 7 (11) 
15(24) \ (7) 29 (48), 8(13) 



7(35) 
7 (39) 



5(33) 
8(53) 



6 (30) 
5(28) 



3 (20) 
2(13) 



Number Standard 
Responding Menn Deviation 



62 
61 



20 
18 



15 
15 



3,2 
3,3 



1,1 
1,2 



1.0 
1.1 



1.1 

,9 



Florida 
Before 
After 



2 (22) 



5(46) 
4 (44) 



4 (36) 
3 (33) 



11 

9 



,8 
,8 



Illinois 
Before 
After 

Montana 
Before 
After 



1 (6) 

2 (14) 



5(19) 
1 (4) 



3(19) 
.2 (14) 



9 (35) 
6(26) 



6 (38) 
4 (29 



5 f!9)i 
3 (13)1 



5(31) 
3(21) 



5(19) 
10 (44) 



1 (6) 
3(21) 



2 (8) 
3(13) 



16 
14 



26 
23 



1,0 

U 



1.2 

1,2 



TABLE 4A (Cont.) 
Opinions of MCT Before and After the Clarification Process 





Strongly 
Oppose 
(1) 


Oppose 
(2) 


Neutral 
(3) 


Favor 
(4) 


Strongly 
Favor 
(5) 


Number 
Responding 


Mean 


Standard 
' Deviation ■ 






H CO 


;n(%) 


H (%) 


H (%) 


H CO 








>')''■■■} 


are 
er 


1 (9) 


3 (21) 
1 (9) , 


l' (7) 


7 (50) 
6 (55) 


3 (21) 
3 (27) 


14 
11 


i v 

3.7 
3.8 


.1.1 
1.3 




sin 
ore 
er 




1 (5) 


6 (23) 
3 (14) 


^3 (50) 
IK(59) 


7 (27) 
5 (23) 


26 
22 


4.0 
.4.0 


J- 

.7 
.8 




* 

I Interest . r 














j 




ps 

ore 

er 


, 1 (14) 
1 (13) 


3 (43) 
2 (25) 


1 (14) 
4 (50) 


2 (29) 


1 (13) 


7 
8 


2.6 
2.8 


i.i 

1.2^ 




ITES 

ore 7 (5) ( 
er - 6 (5) 


23 (17) 
15 (13) 


30 (22) 
23 (19) 


49 (36) 
51 (43) 


26 (19) 
25 (21) 


135 
120 


3.5 
3.6 


1.1 
1.1 

t 























\ 



190 
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■i ' APPENDIX E 

TABLE 4D 

Use of MCT Clarification Process Information for Different Purposes 





Not Useful 
' (1) 


Useful 
(2) 


Highly Useful 
(3) 

NJX) 


Number 
Responding 

— 


Mean 


Standard 
Deviation 

\ ' 





NGS 
















luting Policy 


7 (12) 


in 

30 (51) 


21 (Jo) 


CA 

r" 


i i 
Z.J 


♦ 7 


\ 


ting 

lenentatipn 


20 (35) 


26 (46) 


10 (18) 


57 


1.9 


•8 




mlng Public 


7 -(12) 


' 28 (46) , 


25 (41) 


61 


2.3 


s 

.7 




\ i 

tlatlng Policy 
















i 

Angeles, CA 


4 (24) 


7 (41) 


6 (35) 


17 


2.1 


' .8 




Francisco, CA 


4 (27) 


o (oUJ 




1 C 
1 J 


3 Q, 

t ■ 7/ 


A 
i u 




Ida 


4 (4(0 


6 (60) 




10 


1.6 


.5 




nois 




8 (57) 


6 (43) 




2.4 


.5 


i 

1- 


ana 


1 (4) 


18 (72) 


6 (24) 


25 


2.2 


.5 




a 

onsln 

j , 


1 (5) 


7 (64) 
13 (59) 


4 (36) 
8 (36) 


> 11 

* 

22 


2.4 
2.3 


• 5 
.6 




ial Interest 
oups 


1 (13) 


7 (88) 




8 


1.9 


.4 




a SITES 


15 (12) 


75 (62) 


3H26) 


.122 


2.1 


/ 19.1 
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TABLE 4D (Cont.) 

Use of MCT Clarification Procaas Information for Different Purpoaea 





Not Useful 
(1) 

LSI) 


■ 

Useful 
(2) 


Highly Useful 
(3) 

H (X) 


Number 
Responding 


Mean 


Standard 
Deviation 


sting Ioplementation 














Ansel £8* CA 






1 (6) 


16 


1.8 




Francisco, CA 


11 (73j 


4 (27) 




15 


1.3 


,5 . 


rida 


6 (60) 


4 (40) 




10 

\ 


1.4 


.5 


Lnois 


3 (21) 


9 (64) 


2 (14) 


\ 

14 


1.9 


.6 


tana 


7 (29) 


12 (50) 


5 (21) 


24 


1.9 


•7' ; 


IB 

t 


2 (18) 


7 (64) 


2 (18) 


11 


2.0 


* .6 


:onsin 


8 (38) 


12 (57) 


1 (5) 


21 


1.7 


.6 


*lal Interpnf 
roups 


} (63) • 


3 (38) 




8 


> ( 


.5 


Cj sites 


47 (40) 


61 (51) 


11 (9) 


119 


1.7 v 




laing Public 

/. 

Angeles, CA 














5 (31) , 


9 (56) 


2 (13) - 


16 


1.8 


•7 


Franc isco, CA 


5 (39) 


3 (23) 


5 (39) 


13 


2.0 


•9 


Ida 


2(20) 


. 5 (50) 


•3 (30) 


10 


2.1 


. .7' 


noia 




9 (64) 


5(36) 


14 


2.4 


• .5 



195 



TABLE 4D (Cont.) 

Use of MCT Clarification Process Information for Different Purposes 





Not Useful 
(1) 

■ w 


Useful 
(2) 

N (X) 


Highly Useful 
(3) 

N {Xj 


Number 
Responding 

• 


Mean i 


Standard 
Deviation 


>ntana 




15 (63) 


9 (38) 


24 


2.4 


.5 


axas 


2 (18) 


5 (46) 


4 (36) 


11 


2.2 


.8. 


Lac on sin 


1 (5) 


13 (59) 


8 (36) 


22 


2.3 


.6 


pedal Interest 
Groups 


- 


6 (75) 


2 (25) 


8 


2.3 


.5 

\ 


TAL SITES 


15 (13) 


' 65 (55) 


38 (32) 


118 


2.2 


• 6 


vising or Dropping 














as Angeles, CA 


4 (22) 


11 (61) 


3(17) 


18 


1.9 


c 

• 0 , 


an Francisco i CA 


7 (47) 


6 (40) 


2 (13) 


15 


1.7 


.7 


lor Ida 


6 (60) 


4 (40) 




10 


1.4 


.5 


Uinois 




10 (71) 


4 (29) 


14 


2.3 


•5 


ontana 




22 (92) 


2 (8) 


24 


2.1 


.3 


exas 


3 (30) 


6 (60) 


1 (10) 


10 


1.8 


.6 


isconsin \ 


10 (48) 


10 (48) 


1 (5) 


21 


1.6 


.6 


pecial Interest 
Groups 


1 (13) 


7 (88) 




8 


1.9 


,k ' k 


ITAL SITES 


31 (26) 


76 (63) 


13 (11) 


120 
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APPENDIX E 
TABLE 5 

Usefulness of User's Guide for Different Purposes 



Policy 



on 



ic 



Little Use 
(1) 



Useful 
(2) 



Highly Useful 
(3) 





5 (83) 


1 (17) 




8 (73) 


3 (27) 




11 (79) 


3 (21) 


1 (10) 


4 (40) 


5 (50) 


4 (25) 


10 (63) 


2 (13) 


5 (9) 


38 (67) 


14 (24) 


1 (17) 


4 (67) 


1 (17) 




8 (73) 


3 (27) 




12 (86) 


2/(14) 


2 (20) 


5 (50) 


3 (30) 


2 (14) 


11 (79) 


1 (7) 


5 (9) 


40 (73) 


10 (18) 



1 (17) 


3 (50) 


2 (33) 


4 (33) 


4 (33) 


4 (33) 


2 (13) 


9 (60) 


4 (27) 


3 (30) 


5 (50) 


2 (20) 


6 (40) 


i 6 (40) 


3 (20) 


16 (27) 


1 29 (48) 


15 (25) 



; uaber Stsndsrd 
Responding Mean Deviation 



6 
11 
14 
10 
16 
57 



6 
11 

14 
10 

14 

55 

6 
12 
15 
10 
15 
60 



2.1 



.6 



2.1 



.5 



2.0 



.7 
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